为什么 Pytorch Dropout 层会影响所有值，而不仅仅是设置为零的值?

问题描述

来自 Pytorch 的 dropout 层更改了未设置为零的值.使用 Pytorch 的文档示例:(source):

The dropout layer from Pytorch changes the values that are not set to zero. Using Pytorch's documentation example: (source):

import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.ones(5, 5)

print(input)
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])

然后我通过一个dropout层:

output = m(input)
print(output)
tensor([[0., 0., 2., 2., 0.],
        [2., 0., 2., 0., 0.],
        [0., 0., 0., 0., 2.],
        [2., 2., 2., 2., 2.],
        [2., 0., 0., 0., 2.]])

未设置为零的值现在是 2.为什么?

The values that aren't set to zero are now 2. Why?

推荐答案

这就是 dropout 正则化的工作原理.在 dropout 之后，这些值除以保持概率(在这种情况下为 0.5).

It is how the dropout regularization works. After a dropout the values are divided by the keeping probability (in this case 0.5).

由于 PyTorch Dropout 函数接收将神经元归零的概率作为输入，如果您使用 nn.Dropout(p=0.2) 这意味着它有 0.8 的机会保留.所以表上的值将是 1/(1-0.2).

Since PyTorch Dropout function receives the probability of zeroing a neuron as input, if you use nn.Dropout(p=0.2) that means it has 0.8 chance of keeping. so the values on the table will be 1/(1-0.2).

这被称为反向 dropout 技术"，这样做是为了确保激活的预期值保持不变.

This is called "inverted dropout technique" and it is done in order to ensure that the expected value of the activation remains the same.

这篇关于为什么 Pytorch Dropout 层会影响所有值，而不仅仅是设置为零的值?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！