问题描述
我在keras中训练了一个模型，并且正在考虑修剪我的全连接网络.我对如何修剪图层有点迷失了.
I trained a model in keras and I'm thinking of pruning my fully connected network. I'm little bit lost on how to prune the layers.
有效学习权重和连接方式"一书的作者神经网络公司说，他们为图层的阈值权重添加了一个掩码.我可以尝试做同样的事情，并微调训练好的模型.但是，它如何减少模型大小和计算数量呢?
Author of 'Learning both Weights and Connections for EfficientNeural Networks', say that they add a mask to threshold weights of a layer. I can try to do the same and fine tune the trained model. But, how does it reduce model size and # of computations?
推荐答案
基于评论中的讨论，这是一种修剪神经网络层(权重矩阵)的方法.该方法的本质是根据其范数选择k%最小权重(矩阵的元素)，并将其设置为零.这样，可以将相应的矩阵视为稀疏矩阵，并且我们可以执行密集稀疏矩阵乘法，如果修剪了足够的权重，该乘法会更快.
Based on the discussion in the comments, here is a way to prune a layer (a weight matrix) of your neural network. What the method essentially does is selects the k% smallest weights (elements of the matrix) based on their norm, and sets them to zero. That way, the corresponding matrix can be treated as a sparse matrix, and we can perform dense-sparse matrix multiplication which can be faster if enough weights are pruned.
def weight_pruning(w: tf.Variable, k: float) -> tf.Variable:
"""Performs pruning on a weight matrix w in the following way:
- The absolute value of all elements in the weight matrix are computed.
- The indices of the smallest k% elements based on their absolute values are selected.
- All elements with the matching indices are set to 0.
Args:
w: The weight matrix.
k: The percentage of values (units) that should be pruned from the matrix.
Returns:
The unit pruned weight matrix.
"""
k = tf.cast(tf.round(tf.size(w, out_type=tf.float32) * tf.constant(k)), dtype=tf.int32)
w_reshaped = tf.reshape(w, [-1])
_, indices = tf.nn.top_k(tf.negative(tf.abs(w_reshaped)), k, sorted=True, name=None)
mask = tf.scatter_nd_update(tf.Variable(tf.ones_like(w_reshaped, dtype=tf.float32), name="mask", trainable=False), tf.reshape(indices, [-1, 1]), tf.zeros([k], tf.float32))
return w.assign(tf.reshape(w_reshaped * mask, tf.shape(w)))
虽然上面的方法修剪单个连接(权重)，但是下面的方法从重量矩阵修剪整个神经元.即，该方法基于欧几里得范数选择k%个最小神经元(权重矩阵的列)，并将其设置为零.
While the method above prunes a single connection (weight), the method below prunes a whole neuron from a weight matrix. Namely, the method select the k% smallest neurons (columns of the weight matrix) based on the Euclidean norm, and sets them to zero.
def unit_pruning(w: tf.Variable, k: float) -> tf.Variable:
"""Performs pruning on a weight matrix w in the following way:
- The euclidean norm of each column is computed.
- The indices of smallest k% columns based on their euclidean norms are selected.
- All elements in the columns that have the matching indices are set to 0.
Args:
w: The weight matrix.
k: The percentage of columns that should be pruned from the matrix.
Returns:
The weight pruned weight matrix.
"""
k = tf.cast(
tf.round(tf.cast(tf.shape(w)[1], tf.float32) * tf.constant(k)), dtype=tf.int32
)
norm = tf.norm(w, axis=0)
row_indices = tf.tile(tf.range(tf.shape(w)[0]), [k])
_, col_indices = tf.nn.top_k(tf.negative(norm), k, sorted=True, name=None)
col_indices = tf.reshape(
tf.tile(tf.reshape(col_indices, [-1, 1]), [1, tf.shape(w)[0]]), [-1]
)
indices = tf.stack([row_indices, col_indices], axis=1)
return w.assign(
tf.scatter_nd_update(w, indices, tf.zeros(tf.shape(w)[0] * k, tf.float32))
)
最后，该 Github存储库通过此处介绍的修剪方法并对MNIST数据集进行实验
Finally, this Github repository goes through the pruning methods explained here and performs experiments on the MNIST dataset.
这篇关于如何实现神经网络修剪?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！