问题描述
我正在尝试训练一个非常大的模型.因此,我只能将非常小的批处理大小放入GPU内存中.使用小批量生产的结果非常嘈杂的梯度估计.
我该怎么做才能避免这个问题?
I am trying to train a very large model. Therefore, I can only fit a very small batch size into GPU memory. Working with small batch sizes results with very noisy gradient estimations.
What can I do to avoid this problem?
推荐答案
您可以在求解器参数中更改iter_size
.Caffe在每个随机梯度下降步骤中的iter_size
x batch_size
实例上累积梯度.因此,当由于内存有限而无法使用大的batch_size时,增加iter_size
也会获得更稳定的渐变.
You can change the iter_size
in the solver parameters.Caffe accumulates gradients over iter_size
x batch_size
instances in each stochastic gradient descent step.So increasing iter_size
can also get more stable gradient when you cannot use large batch_size due to the limited memory.
这篇关于Caffe:如果只有一小部分内存可以怎么办?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!