问题描述
我正在为 deep-q 网络实现优先体验重放,规范的一部分是将梯度乘以所谓的重要性采样 (IS) 权重.以下论文的第 3.4 节讨论了梯度修改:https://arxiv.org/pdf/1511.05952.pdf 我正在努力创建一个自定义损失函数,除了 y_true
和 y_pred
之外,它还接受一系列 IS 权重.
I'm working on implementing prioritized experience replay for a deep-q network, and part of the specification is to multiply gradients by what's know as importance sampling (IS) weights. The gradient modification is discussed in section 3.4 of the following paper: https://arxiv.org/pdf/1511.05952.pdf I'm struggling with creating a custom loss function that takes in an array of IS weights in addition to y_true
and y_pred
.
这是我的模型的简化版本:
Here's a simplified version of my model:
import numpy as np
import tensorflow as tf
# Input is RAM, each byte in the range of [0, 255].
in_obs = tf.keras.layers.Input(shape=(4,))
# Normalize the observation to the range of [0, 1].
norm = tf.keras.layers.Lambda(lambda x: x / 255.0)(in_obs)
# Hidden layers.
dense1 = tf.keras.layers.Dense(128, activation="relu")(norm)
dense2 = tf.keras.layers.Dense(128, activation="relu")(dense1)
dense3 = tf.keras.layers.Dense(128, activation="relu")(dense2)
dense4 = tf.keras.layers.Dense(128, activation="relu")(dense3)
# Output prediction, which is an action to take.
out_pred = tf.keras.layers.Dense(2, activation="linear")(dense4)
opt = tf.keras.optimizers.Adam(lr=5e-5)
network = tf.keras.models.Model(inputs=in_obs, outputs=out_pred)
network.compile(optimizer=opt, loss=huber_loss_mean_weighted)
这是我的自定义损失函数,它只是一个 Huber 损失乘以 IS 权重的实现:
Here's my custom loss function, which is just an implementation of Huber Loss multiplied by the IS weights:
'''
' Huber loss: https://en.wikipedia.org/wiki/Huber_loss
'''
def huber_loss(y_true, y_pred):
error = y_true - y_pred
cond = tf.keras.backend.abs(error) < 1.0
squared_loss = 0.5 * tf.keras.backend.square(error)
linear_loss = tf.keras.backend.abs(error) - 0.5
return tf.where(cond, squared_loss, linear_loss)
'''
' Importance Sampling weighted huber loss.
'''
def huber_loss_mean_weighted(y_true, y_pred, is_weights):
error = huber_loss(y_true, y_pred)
return tf.keras.backend.mean(error * is_weights)
重要的一点是 is_weights
是动态的,即每次 fit()
被调用时它都是不同的.因此,我不能简单地关闭 is_weights
如下所述:在keras中制作自定义损失函数
The important bit is that is_weights
is dynamic, i.e. it's different each time fit()
is called. As such, I cannot simply close over is_weights
as described here: Make a custom loss function in keras
我在网上找到了这段代码,它似乎使用了 Lambda
层来计算损失:https://github.com/keras-team/keras/blob/master/examples/image_ocr.py#L475 看起来很有希望,但我我正在努力理解它/使其适应我的特定问题.任何帮助表示赞赏.
I found this code online, which appears to use a Lambda
layer to compute the loss: https://github.com/keras-team/keras/blob/master/examples/image_ocr.py#L475 It looks promising, but I'm struggling to understand it/adapt it to my particular problem. Any help is appreciated.
推荐答案
好的.这是一个例子.
from keras.layers import Input, Dense, Conv2D, MaxPool2D, Flatten
from keras.models import Model
from keras.losses import categorical_crossentropy
def sample_loss( y_true, y_pred, is_weight ) :
return is_weight * categorical_crossentropy( y_true, y_pred )
x = Input(shape=(32,32,3), name='image_in')
y_true = Input( shape=(10,), name='y_true' )
is_weight = Input(shape=(1,), name='is_weight')
f = Conv2D(16,(3,3),padding='same')(x)
f = MaxPool2D((2,2),padding='same')(f)
f = Conv2D(32,(3,3),padding='same')(f)
f = MaxPool2D((2,2),padding='same')(f)
f = Conv2D(64,(3,3),padding='same')(f)
f = MaxPool2D((2,2),padding='same')(f)
f = Flatten()(f)
y_pred = Dense(10, activation='softmax', name='y_pred' )(f)
model = Model( inputs=[x, y_true, is_weight], outputs=y_pred, name='train_only' )
model.add_loss( sample_loss( y_true, y_pred, is_weight ) )
model.compile( loss=None, optimizer='sgd' )
print model.summary()
注意,由于您已经通过add_loss()
添加了损失,因此您不必通过compile( loss=xxx )
来添加.
Note, since you've add loss through add_loss()
, you don't have to do it through compile( loss=xxx )
.
关于训练模型,除了将 y_true
移动到输入端之外,没有什么特别的.见下文
With regards to train a model, nothing is special except you move y_true
to your input end. See below
import numpy as np
a = np.random.randn(8,32,32,3)
a_true = np.random.randn(8,10)
a_is_weight = np.random.randint(0,2,size=(8,1))
model.fit( [a, a_true, a_is_weight] )
最后,您可以制作一个测试模型(共享 model
中的所有权重)以便于使用,即
Finally, you can make a testing model (which share all weights in model
) for easier use, i.e.
test_model = Model( inputs=x, outputs=y_pred, name='test_only' )
a_pred = test_model.predict( a )
这篇关于具有附加动态参数的 Keras 损失函数的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!