我想在TensorFlow中使用批处理规范化。我在 core/ops/nn_ops.cc 中找到了相关的C++源代码。但是,我没有在tensorflow.org上找到它的文档。

BN在MLP和CNN中具有不同的语义,因此我不确定此BN的确切作用。

我也没有找到称为MovingMoments的方法。

最佳答案

更新2016年7月在TensorFlow中使用批处理规范化的最简单方法是通过contrib/layerstflearnslim中提供的更高级别的接口(interface)。

如果要DIY ,请使用上一个答案:
自发布以来,此文档的文档字符串已得到改进-请参见docs comment in the master branch而不是找到的。它特别说明了它是tf.nn.moments的输出。

您可以在batch_norm test code中看到一个非常简单的示例。对于更真实的使用示例,我将其包含在帮助器类下面,并使用了我为自己使用而写的注释(不提供保修!):

"""A helper class for managing batch normalization state.

This class is designed to simplify adding batch normalization
(http://arxiv.org/pdf/1502.03167v3.pdf) to your model by
managing the state variables associated with it.

Important use note:  The function get_assigner() returns
an op that must be executed to save the updated state.
A suggested way to do this is to make execution of the
model optimizer force it, e.g., by:

  update_assignments = tf.group(bn1.get_assigner(),
                                bn2.get_assigner())
  with tf.control_dependencies([optimizer]):
    optimizer = tf.group(update_assignments)

"""

import tensorflow as tf


class ConvolutionalBatchNormalizer(object):
  """Helper class that groups the normalization logic and variables.

  Use:
      ewma = tf.train.ExponentialMovingAverage(decay=0.99)
      bn = ConvolutionalBatchNormalizer(depth, 0.001, ewma, True)
      update_assignments = bn.get_assigner()
      x = bn.normalize(y, train=training?)
      (the output x will be batch-normalized).
  """

  def __init__(self, depth, epsilon, ewma_trainer, scale_after_norm):
    self.mean = tf.Variable(tf.constant(0.0, shape=[depth]),
                            trainable=False)
    self.variance = tf.Variable(tf.constant(1.0, shape=[depth]),
                                trainable=False)
    self.beta = tf.Variable(tf.constant(0.0, shape=[depth]))
    self.gamma = tf.Variable(tf.constant(1.0, shape=[depth]))
    self.ewma_trainer = ewma_trainer
    self.epsilon = epsilon
    self.scale_after_norm = scale_after_norm

  def get_assigner(self):
    """Returns an EWMA apply op that must be invoked after optimization."""
    return self.ewma_trainer.apply([self.mean, self.variance])

  def normalize(self, x, train=True):
    """Returns a batch-normalized version of x."""
    if train:
      mean, variance = tf.nn.moments(x, [0, 1, 2])
      assign_mean = self.mean.assign(mean)
      assign_variance = self.variance.assign(variance)
      with tf.control_dependencies([assign_mean, assign_variance]):
        return tf.nn.batch_norm_with_global_normalization(
            x, mean, variance, self.beta, self.gamma,
            self.epsilon, self.scale_after_norm)
    else:
      mean = self.ewma_trainer.average(self.mean)
      variance = self.ewma_trainer.average(self.variance)
      local_beta = tf.identity(self.beta)
      local_gamma = tf.identity(self.gamma)
      return tf.nn.batch_norm_with_global_normalization(
          x, mean, variance, local_beta, local_gamma,
          self.epsilon, self.scale_after_norm)

请注意,我将其称为ConvolutionalBatchNormalizer是因为它固定使用tf.nn.moments来在轴0、1和2上求和,而对于非卷积使用,您可能只希望轴0。

如果您使用它,反馈表示赞赏。

10-06 05:20
查看更多