本文主要采摘、翻译、修改自 P. Galeone’s blog Analysis of Dropout。
Overfitting is a problem in Deep Neural Networks (DNN): the model learns to classify only the training set, adapting itself to the training examples instead of learning decision boundaries capable of classifying generic instances. Many solutions to the overfitting problem have been presented during these years; one of them have overwhelmed the others due to its simplicity and its empirical good results: Dropout.
Visual representation of Dropout, right from the paper. On the left there’s the network before applying Dropout, on the right the same network with Dropout applied.The network on the left it’s the same network used at test time, once the parameters have been learned.
The idea behind Dropout is to train an ensemble of DNNs and average the results of the whole ensemble instead of train a single DNN.(Dropout 背后的思想是训练一个含有多个 DNN 的集成学习器。)
The DNNs are built dropping out neurons with ( p ) probability, therefore keeping the others on with probability ( q=1−p ). When a neuron is dropped out, its output is set to zero, no matter what the input or the associated learned parameter is.
The dropped neurons do not contribute to the training phase in both the forward and backward phases of back-propagation: for this reason every time a single neuron is dropped out it’s like the training phase is done on a new network.(被 dropout 的神经元不参与训练过程的前向传播和反向传播，每次进行一个 batch 的训练的时候，由于被 dropout 的神经元组合不同，就像是在训练一个新的神经网络。)
Quoting the authors:
In short: Dropout works well in practice because it prevents the co-adaption of neurons during the training phase.
Now that we got an intuitive idea behind Dropout, let’s analyze it in depth.
How Dropout works
As said before, Dropout turns off neurons with probability ( p ) and therefore let the others turned on with probability ( q=1−p )
Every single neuron has the same probability of being turned off. This means that:
Given
- ( h(x) = Wx + b ) a linear projection of a ( d_i )-dimensional input x into a ( d_h )-dimensional output space.
- ( a(h) ) an activation function
it’s possible to model the application of Dropout, in the training phase only, to the given projection as a modified activation function (可以将 Dropout 建模为一个特殊的激活函数（element-wise）):
[
f(h) = D bigodot a(h)
]
Where ( D=(X_1, …, X_{d_h}) ) is a ( d_h )-dimensional vector of Bernoulli variables ( X_i ) （( d_h ) 维伯努利随机变量的向量）.
伯努利随机变量
A Bernoulli random variable(伯努利随机变量) has the following probability mass distribution（pdf，概率质量函数）:
[
f(k; p) = p^k (1-p)^{(1-k)} =
begin{cases}
p text{ if } k = 1 \\
(1-p) text{ if } k = 0
end{cases}
]
Where k are the possible outcomes.
设 drop rate 为 ( p )。
It’s evident that this random variable perfectly models the Dropout application on a single neuron. In fact, the neuron is turned off with probability ( p = P(X_i = 0) ) and kept on otherwise.
伯努利试验成功的概率是 ( P( X_i = 1) = 1 - p )。伯努利试验成功，则神经元被保留。伯努利试验失败，则神经元被关闭。
It can be useful to see the application of Dropout on a generic i-th neuron:
[
O_i = X_i a(sum_{k=1}^{d_i} w_k x_k +b) = begin{cases}
a(sum_{k=1}^{d_i} w_k x_k +b) text{ if } X_i = 1 \\
0 text{ if } X_i = 0
end{cases}
]
where ( P(X_i = 0) = p ) （即 (X_i thicksim Bernoulli(1-p) )，( X_i ) 服从成功概率为 ( (1-p) ) 的伯努利分布。）
Since during train phase a neuron is kept on with probability ( q ), during the testing phase we have to emulate the behavior of the ensemble of networks used in the training phase.
To do this, the authors suggest scaling the activation function by a factor of ( q ) during the test phase in order to use the expected output produced in the training phase as the single output required in the test phase. Thus:
Train phase:
[
O_i = X_i a(sum_{k=1}^{d_i} w_k x_k + b)
]
Test phase:
[
O_i= q a(sum_{k=1}^{d_i} w_k x_k + b)
]
Inverted Dropout
A slightly different approach is to use Inverted Dropout. This approach consists in the scaling of the activations during the training phase, leaving the test phase untouched.
The scale factor is the inverse of the keep probability: ( frac{1}{1-p} = frac{1}{q} ), thus:
Train phase:
[
O_i = frac{1}{q} X_i a(sum_{k=1}^{di} w_k x_k + b)
]
Test phase:
[
O_i = a(sum_{k=1}^{d_i} w_k x_k + b)
]
Inverted Dropout is how Dropout is implemented in practice in the various deep learning frameworks because it helps to define the model once and just change a parameter (the keep/drop probability) to run train and test on the same model.
Direct Dropout, instead, force you to modify the network during the test phase because if you don’t multiply by ( q ) the output the neuron will produce values that are higher respect to the one expected by the successive neurons (thus the following neurons can saturate or explode): that’s why Inverted Dropout is the more common implementation.
Dropout of a set of neurons
设 drop rate 为 ( p )。
It can be easily noticed that a layer ( h ) with ( n ) neurons, in a single train step, can be seen as an ensemble of ( n ) Bernoulli experiments, each one with a probability of success equals to ( 1 - p ). (可以看成是 ( n ) 次成功概率为 ( 1 - p ) 伯努利试验)。
Thus, the output of the layer h have a number of dropped neurons equals to (被关闭的神经元的数目为):
[
Y = sum_{k=1}^{d_h} 1 - X_i
]
Since every neuron is now modeled as a Bernoulli random variable and all these random variables are independent and identically distributed, the total number of dropped neuron is a random variable too, called Binomial(二项分布):
[
Y thicksim Bi(d_h, p)
]
Where the probability of getting exactly k shutdown neurons in n is given by the probability mass distribution:
[
f(k;n, p) = left( begin{matrix} k \ n end{matrix} right) p^k (1-p)^{(n-k)}
]
We can now use this distribution to analyze the probability of dropping a specified number of neurons.
When using Dropout, we define a fixed Dropout probability ( p ) for a chosen layer and we expect that a proportional number of neurons are dropped from it.
For example, if the layer we apply Dropout to has ( n=1024 ) neurons and ( p=0.5 ), we expect that 512 get dropped. Let’s verify this statement:
[
Y = sum_{i=1}^{1024} X_i \\
Y thicksim Bi(1024, 0.5) \\
P(Y=512) = left( begin{matrix} 512 \ 1024 end{matrix} right) 0.5^{512} times (1−0.5)^{(1024−512)} approx 0.025
]
Thus, the probability of dropping out exactly ( np=512 ) neurons is of only 0.025!
A python 3 script can help us to visualize how neurons are dropped for different values of p and a fixed value of n. The code is commented.
The binomial distribution is very peaked around ( np ). As we can see from the image above no matter what the p value is, the number of neurons dropped on average is proportional to ( np ), in fact:
[
E[Bi(n,p)]=np
]
Moreover, we can notice that the distribution of values is almost symmetric around ( p=0.5 ) and the probability of dropping ( np ) neurons increase as the distance from ( p=0.5 ) increase.
The scaling factor has been added by the authors to compensate(补偿) the activation values, because they expect that during the training phase only a percentage of ( 1−p ) neurons have been kept. During the testing phase, instead, the 100% of neurons are kept on, thus the value should be scaled down accordingly.
Dropout & other regularizers
Dropout is often used with L2 normalization and other parameter constraint techniques (such as Max Norm), this is not a case. Normalizations help to keep model parameters value low, in this way a parameter can’t grow too much.（将权重参数保持在较小的数值。）
In brief, the L2 normalization (for example) is an additional term to the loss, where ( lambda in [0, 1] ) is an hyper-parameter called regularization strength, ( F(W;x) ) is the model and ( varepsilon ) is the error function between the real ( y ) and the predicted ( tilde{y} ).
[
mathcal{L} = varepsilon(y, F(W;x)) + frac{lambda}{2} W^2
]
It’s easy to understand that this additional term, when doing back-propagation via gradient descent, reduces the update amount (使得更新不会正的太大). If ( eta ) is the learning rate, the update amount of the parameter (w in W) is
[
w gets w − eta (frac{partial F(W;x) }{partial w} + lambda w)
]
Dropout alone, instead, does not have any way to prevent parameter values from becoming too large during this update phase. Moreover, the inverted implementation leads the update steps to become bigger, as showed below.
Inverted Dropout and other regularizers
Since Dropout does not prevent parameters from growing and overwhelming each other, applying L2 regularization (or any other regularization technique that constraints the parameter values) can help.
Making explicit the scaling factor, the previous equation becomes:
[
w gets w − eta (frac{1}{q} frac{partial F(W;x) }{partial w} + lambda w)
]
It can be easily seen that when using Inverted Dropout, the learning rate is scaled by a factor of q. q has values in [0,1].For this reason, from now on we’ll call ( q ) boosting factor because it boosts the learning rate. Moreover, we’ll call ( frac{eta}{q} ) the effective learning rate.
The effective learning rate, thus, is higher respect to the learning rate chosen: for this reason normalizations that constrain the parameter values can help to simplify the learning rate selection process.
Summary
- Dropout exists in two versions: direct (not commonly implemented) and inverted
- Dropout on a single neuron can be modeled using a Bernoulli random variable
- Dropout on a set of neurons can be modeled using a Binomial random variable
- Even if the probability of dropping exactly ( np ) neurons is low, ( np ) neurons are dropped on average on a layer of ( n ) neurons.
- Inverted Dropout boost the learning rate
- Inverted Dropout should be using together with other normalization techniques that constrain the parameter values in order to simplify the learning rate selection procedure
- Dropout helps to prevent overfitting in deep neural networks.
- Max Norm impose a constraint to the parameters size. Chosen a value for the hyper-parameter ( c ). It imposes the constraint ( |w| leq c ).