问题描述
我正在执行通过定向反向传播生成热图的任务.我已经覆盖了原始的Relu，并获得了每个参数的梯度.但是，我不确定下一步该怎么做.感谢您的协助！谢谢！
I am working on a task to generate heatmap by guided backpropagation. I have overridden the original Relu and obtained the gradient for each parameter. However, I am not sure what should I do next. Your assistance is appreciated! Thank you!
这是我的代码:
我首先像这样使用@tf.RegisterGradient("GuidedRelu"):
def _GuidedReluGrad(op, grad):
gate_f = tf.cast(op.outputs[0] > 0, "float32")
gate_R = tf.cast(grad > 0, "float32")
return gate_f * gate_R * grad
然后，我通过以下方式获得了毕业成绩:
Then, I obtained grads by:
with g.gradient_override_map({"Relu": "GuidedRelu"}):
with tf.GradientTape() as tape:
logits = self.net(tf.cast(img, dtype=tf.float32))
xentropy = tf.nn.softmax_cross_entropy_with_logits(
labels=tf.cast(
tf.one_hot(predicted_class, depth=1000), dtype=tf.int32
),
logits=logits,
)
reduced = tf.reduce_mean(xentropy)
grads = tape.gradient(reduced, self.net.trainable_variables)
我发现第一层的渐变形状为(7、7、3、64).但我不知道如何使用此grad生成与输入大小相似的热图.
I found the grad for the first layer has shape (7, 7, 3, 64). But I don't know how to use this grad to generate a heatmap that has similar size to the input.
推荐答案
它类似于该层的内核可视化.下面是我可视化具有(7, 7, 4, 4)形状的Conv2D层的示例. (7, 7, 4, 4)表示该层具有7*7 Kernels，其中4 incoming filters(上一层的过滤器)，最后一个4是该层的outgoing filters.
It is similar to kernel visualization of the layer. Below is an example where I am visualizing Conv2D layer having (7, 7, 4, 4) shape. (7, 7, 4, 4) means the layer has 7*7 Kernels with 4 incoming filters(filters of previous layer) and last 4 is the outgoing filters of this layer.
因此，在您的情况下，(7, 7, 3, 64)表示您具有7*7 Kernels和3 incoming filters(因为它是您的第一层，猜测您的输入是彩色图像)，而64是您的层filters.
So in your case, (7, 7, 3, 64) means you have 7*7 Kernels with 3 incoming filters(as it is your first layer, guessing your input is a color image) and 64 is your layers filters.
供我参考，我已经打印了模型的所有卷积层.我在可视化代码中使用相同的代码来获取最后一层的过滤器形状.即conv2d_3 (7, 7, 4, 4)并将其用于可视化-
For your reference, I have print all the Convolution layers of my model. I am using the same code in visualization code to get the filter shapes of last layer .i.e. conv2d_3 (7, 7, 4, 4) and use it for visualization -
# summarize filter shapes
for layer in model.layers:
# check for convolutional layer
if 'conv' in layer.name:
# get filter weights
filters, biases = layer.get_weights()
print(layer.name, filters.shape)
输出-
conv2d_1 (3, 3, 3, 2)
conv2d_2 (3, 3, 2, 4)
conv2d_3 (7, 7, 4, 4)
我们将可视化conv2d_3 (7, 7, 4, 4)，因为它与您的要求类似.因此，基本上我们应该有(incoming filters * outgoing filters = 16)16个大小为7*7的图像.
We shall Visualize conv2d_3 (7, 7, 4, 4) as it is similar to your requirement. So basically we should have (incoming filters * outgoing filters = 16)16 images of 7*7 size.
可视化代码-您需要分别修改incoming_filters和outgoing_filters分别是上一层的过滤器(如果是第一层，则是图像的通道大小)和该层的过滤器.
Code for Visualization - You need to modify incoming_filters and outgoing_filters which are filters of previous layer(if it is first layer then the channel size of image) and filters of this layer respectively.
from matplotlib import pyplot
# filters will have details of last Conv layer .i.e. conv2d_3 (7, 7, 4, 4)
for layer in model.layers:
# check for convolutional layer
if 'conv' in layer.name:
# get filter weights
filters, biases = layer.get_weights()
# Fix the figure size
fig, ax = pyplot.subplots(figsize=(15, 15))
# Normalize filter values to 0-1 so we can visualize them
f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)
outgoing_filters, ix = 4, 1
for i in range(outgoing_filters):
# get the filter
f = filters[:, :, :, i]
# plot each channel separately
incoming_filters = 4
for j in range(incoming_filters):
# specify subplot and turn of axis
ax = pyplot.subplot(incoming_filters, outgoing_filters, ix)
ax.set_xticks([])
ax.set_yticks([])
# plot filter channel
# Use cmap='gray' for Gray scale image
pyplot.imshow(f[:, :, j])
ix += 1
# show the figure
pyplot.show()
输出-
希望这能回答您的问题.学习愉快.
Hope this answers your question. Happy Learning.
编辑-花费了更多精力在每个时期之后捕获渐变并将其可视化.在下面的代码中使用，以在每个时期之后捕获渐变.我正在使用旧方法在Tensorflow 1.15.0中捕获渐变，而不是使用tf.GradientTape.如果您想了解如何使用tf.GradientTape捕获梯度，则可以参考我们对此.
Edit - Made some more effort to capture gradients after every epoch and visualize them. Used below code to capture gradients after every epoch. I am using old way to capture gradients in Tensorflow 1.15.0 and not using tf.GradientTape. If you would like to know How to capture gradient using tf.GradientTape then you can refer our answer to this question.
在下面的程序中，gradient是array，具有每个层的每个历元之后捕获的渐变.
In the below program, gradient is the array that has gradients captured after every epoch for every layer.
代码-
# (1) Importing dependency
%tensorflow_version 1.x
import tensorflow as tf
import keras
from keras import backend as K
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Flatten, Conv2D, MaxPooling2D, Conv3D
from keras.layers.normalization import BatchNormalization
import numpy as np
np.random.seed(1000)
# (2) Get Data
import tflearn.datasets.oxflower17 as oxflower17
x, y = oxflower17.load_data(one_hot=True)
# (3) Create a sequential model
model = Sequential()
# 1st Convolutional Layer
model.add(Conv2D(filters=2, input_shape=(224,224,3), kernel_size=(3,3), strides=(4,4), padding='Same'))
model.add(Activation('relu'))
# 2nd Convolutional Layer
model.add(Conv2D(filters=4, kernel_size=(3,3), strides=(1,1), padding='Same'))
model.add(Activation('relu'))
# 3rd Convolutional Layer
model.add(Conv2D(filters=4, kernel_size=(7,7), strides=(1,1), padding='Same'))
model.add(Activation('relu'))
# Passing it to a dense layer
model.add(Flatten())
# 1st Dense Layer
model.add(Dense(100))
model.add(Activation('relu'))
# Output Layer
model.add(Dense(17))
model.add(Activation('softmax'))
model.summary()
# (4) Compile
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
epoch_gradient = []
def get_gradient_func(model):
grads = K.gradients(model.total_loss, model.trainable_weights)
inputs = model.model._feed_inputs + model.model._feed_targets + model.model._feed_sample_weights
func = K.function(inputs, grads)
return func
# Define the Required Callback Function
class GradientCalcCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
get_gradient = get_gradient_func(model)
grads = get_gradient([x, y, np.ones(len(y))])
epoch_gradient.append(grads)
epoch = 4
model.fit(x, y, batch_size=64, epochs= epoch, verbose=1, validation_split=0.2, shuffle=True, callbacks=[GradientCalcCallback()])
# (7) Convert to a 2 dimensiaonal array of (epoch, gradients) type
gradient = np.asarray(epoch_gradient)
print("Total number of epochs run:", epoch)
print("Gradient Array has the shape:",gradient.shape)
输出-
TensorFlow 1.x selected.
Using TensorFlow backend.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/helpers/summarizer.py:9: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/helpers/trainer.py:25: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/collections.py:13: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/config.py:123: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/config.py:129: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tflearn/config.py:131: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
Downloading Oxford 17 category Flower Dataset, Please wait...
100.0% 60276736 / 60270631
('Succesfully downloaded', '17flowers.tgz', 60270631, 'bytes.')
File Extracted
Starting to parse images...
Parsing Done!
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 56, 56, 2) 56
_________________________________________________________________
activation_1 (Activation) (None, 56, 56, 2) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 56, 56, 4) 76
_________________________________________________________________
activation_2 (Activation) (None, 56, 56, 4) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 56, 56, 4) 788
_________________________________________________________________
activation_3 (Activation) (None, 56, 56, 4) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 12544) 0
_________________________________________________________________
dense_1 (Dense) (None, 100) 1254500
_________________________________________________________________
activation_4 (Activation) (None, 100) 0
_________________________________________________________________
dense_2 (Dense) (None, 17) 1717
_________________________________________________________________
activation_5 (Activation) (None, 17) 0
=================================================================
Total params: 1,257,137
Trainable params: 1,257,137
Non-trainable params: 0
_________________________________________________________________
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:431: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:438: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
Train on 1088 samples, validate on 272 samples
Epoch 1/4
1088/1088 [==============================] - 5s 5ms/step - loss: 2.8055 - accuracy: 0.0846 - val_loss: 2.7566 - val_accuracy: 0.1176
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Epoch 2/4
1088/1088 [==============================] - 5s 5ms/step - loss: 2.3974 - accuracy: 0.3263 - val_loss: 2.5707 - val_accuracy: 0.2132
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Epoch 3/4
1088/1088 [==============================] - 5s 5ms/step - loss: 1.5953 - accuracy: 0.5506 - val_loss: 2.4076 - val_accuracy: 0.2684
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Epoch 4/4
1088/1088 [==============================] - 5s 5ms/step - loss: 0.8699 - accuracy: 0.7812 - val_loss: 2.5698 - val_accuracy: 0.3162
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py:111: UserWarning: `Sequential.model` is deprecated. `Sequential` is a subclass of `Model`, you can just use your `Sequential` instance directly.
warnings.warn('`Sequential.model` is deprecated. '
Total number of epochs run: 4
Gradient Array has the shape: (4, 10)
可视化-
让可视化gradient[0][4]，即[0]表示模型的第一个纪元，[4]表示模型的第5个纪元.
Lets Visualize gradient[0][4], .i.e. [0] means the 1st epoch and [4] means fifth later of the model.
from matplotlib import pyplot
filters = gradient[0][4]
# Fix the figure size
fig, ax = pyplot.subplots(figsize=(15, 15))
# Normalize filter values to 0-1 so we can visualize them
f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)
outgoing_filters, ix = 4, 1
for i in range(outgoing_filters):
# get the filter
f = filters[:, :, :, i]
# plot each channel separately
incoming_filters = 4
for j in range(incoming_filters):
# specify subplot and turn of axis
ax = pyplot.subplot(incoming_filters, outgoing_filters, ix)
ax.set_xticks([])
ax.set_yticks([])
# plot filter channel
# Use cmap='gray' for Gray scale image
pyplot.imshow(f[:, :, j])
ix += 1
# show the figure
pyplot.show()
输出-
如果您想可视化Conv3D，请参考此 answer .
If you would like to Visualize Conv3D then refer this answer.
希望这可以详细回答您的问题.学习愉快.
Hope this answers your question in detail. Happy Learning.
这篇关于在Tensorflow 2中将梯度可视化为热图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！