我正在尝试在计算渐变体面中实现mxnet in this page的教程:

def SGD(params, lr):
for param in params:
    param[:] = param - lr * param.grad


我注意到当lr
lr * param.grad


变为具有相同param.grad大小的零矩阵

我不知道为什么会这样。谁能帮助我理解这一点?

非常感谢

最佳答案

我在本地的PyCharm中从该笔记本运行了代码,并附加了调试器以尝试重现您看到的内容。

def SGD(params, lr):
    for param in params:
        t = lr * param.grad
        param[:] = param - t


当我在上面的代码中放置一个断点并检查t时,调试器显示所有零。但是,当我运行代码时,一切运行正常。那很奇怪。因此,我修改了代码以在每次迭代中打印每个像素的所有梯度之和。

def SGD(params, lr):
    for param in params:
        t = lr * param.grad
        print(nd.sum(param.grad, axis=-1))
        param[:] = param - t


我得到了this输出。一次迭代的输出如下所示:

[  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -6.23335197e-08
   2.91620381e-08   2.32830644e-10   1.11467671e-08  -7.45058060e-09
   2.09547579e-09  -3.11993062e-08   3.44589353e-08  -7.45058060e-09
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   1.73723835e-09  -1.45343968e-08   1.51213424e-08  -4.47034836e-08
  -1.19209290e-07  -8.10250640e-08   8.66129994e-08   3.77185643e-08
   8.47503543e-08   8.66129994e-08   8.70786607e-08   5.35510480e-09
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   5.47231149e-09  -4.99075004e-10  -4.99075004e-10
   6.05359674e-09  -9.66247171e-09   1.83936208e-08  -7.50416707e-09
   1.12872200e-07   9.31513569e-08   2.45261617e-07  -1.86264515e-07
  -5.28991222e-07  -2.68220901e-07  -1.22934580e-07   0.00000000e+00
   2.08616257e-07   1.89989805e-07   4.33064997e-08   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00  -2.75546541e-09   2.39708999e-08  -2.04163371e-08
   6.70552254e-08   6.14672899e-08   4.40049917e-08   8.94069672e-08
   8.94069672e-08   2.98023224e-07   1.19209290e-07  -5.96046448e-08
  -2.38418579e-07  -4.76837158e-07  -4.76837158e-07  -2.38418579e-07
   1.78813934e-07   4.13507223e-07   2.83122063e-07   3.09199095e-07
   1.34285074e-07   6.17840215e-08   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00  -2.19943086e-09   1.58324838e-08   6.26314431e-08
   4.47034836e-08   5.96046448e-08   2.38418579e-07   4.76837158e-07
   2.38418579e-07   4.76837158e-07   5.66244125e-07   1.49011612e-07
  -4.17232513e-07  -2.38418579e-07  -3.57627869e-07   0.00000000e+00
   0.00000000e+00   1.22934580e-07   1.97440386e-07   2.73808837e-07
   4.55183908e-08   1.16733858e-08   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   7.33416528e-09   4.35829861e-08
   4.36230039e-08   3.72456270e-08   1.67638063e-08   5.96046448e-08
   1.49011612e-07   2.38418579e-07  -1.19209290e-07  -2.38418579e-07
  -1.78813934e-07   2.38418579e-07   6.25848770e-07   3.57627869e-07
   2.98023224e-08  -4.17232513e-07  -3.57627869e-07  -5.96046448e-08
  -1.04308128e-07   1.19209290e-07   1.08033419e-07  -6.14672899e-08
  -1.40164047e-07  -1.22236088e-07  -1.07334927e-07   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -2.17623892e-08   6.13508746e-08
   7.71033228e-08   4.65661287e-09   0.00000000e+00  -5.96046448e-08
   0.00000000e+00   2.38418579e-07   1.19209290e-07   3.57627869e-07
   4.76837158e-07   1.19209290e-07  -2.38418579e-07  -1.78813934e-07
  -4.76837158e-07   0.00000000e+00   1.78813934e-07   1.19209290e-07
   5.06639481e-07   1.78813934e-07  -1.19209290e-07  -1.19209290e-07
  -1.19209290e-07   0.00000000e+00  -1.34168658e-08   0.00000000e+00
   0.00000000e+00   0.00000000e+00   4.71118256e-10   2.32830644e-09
   2.53785402e-08  -2.83122063e-07   1.19209290e-07   1.19209290e-07
   2.98023224e-07   0.00000000e+00   2.38418579e-07   3.20374966e-07
   0.00000000e+00   3.57627869e-07  -2.38418579e-07   2.38418579e-07
  -9.53674316e-07  -7.15255737e-07   1.19209290e-07   3.57627869e-07
   4.17232513e-07  -7.45058060e-08  -5.96046448e-08  -1.19209290e-07
   1.49011612e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -2.59024091e-09
  -5.21540642e-08  -7.45058060e-09  -1.78813934e-07   5.96046448e-08
   1.78813934e-07   1.19209290e-07   4.76837158e-07   2.98023224e-07
  -5.96046448e-08   2.38418579e-07   3.57627869e-07   5.96046448e-08
  -4.76837158e-07   1.78813934e-07   4.17232513e-07  -1.19209290e-07
   2.98023224e-07   5.96046448e-08   2.04890966e-08  -1.30385160e-08
  -4.83123586e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   3.23634595e-08
  -5.58793545e-08  -1.34110451e-07  -1.49011612e-08  -5.96046448e-08
   1.49011612e-07   1.49011612e-08   1.78813934e-07   0.00000000e+00
   0.00000000e+00   8.94069672e-08   5.96046448e-08  -1.19209290e-07
  -3.57627869e-07   0.00000000e+00   4.76837158e-07  -2.38418579e-07
   0.00000000e+00   1.02445483e-07   1.46217644e-07  -1.32713467e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -2.79396772e-08  -1.56462193e-07  -1.93715096e-07   1.82539225e-07
   3.68803740e-07   4.32133675e-07   5.96046448e-08  -1.19209290e-07
   0.00000000e+00   2.98023224e-07  -1.19209290e-07  -5.06639481e-07
   1.19209290e-07   4.76837158e-07   8.94069672e-08   0.00000000e+00
   0.00000000e+00   5.96046448e-08   8.61473382e-08   4.39467840e-08
   4.85442797e-11   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -2.79396772e-08  -1.86264515e-07  -1.93715096e-07   2.98023224e-08
   3.57627869e-07   3.57627869e-07  -2.38418579e-07  -1.19209290e-07
  -2.38418579e-07  -1.78813934e-07  -1.78813934e-07   0.00000000e+00
  -4.02331352e-07  -2.23517418e-08   2.08616257e-07  -2.98023224e-08
   2.38418579e-07   2.08616257e-07   2.37952918e-07   7.72388375e-08
   3.80787242e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -7.45058060e-08  -2.98023224e-08   1.11758709e-07   1.78813934e-07
   4.76837158e-07   3.57627869e-07  -1.19209290e-07   0.00000000e+00
  -3.57627869e-07   2.38418579e-07  -1.19209290e-07   5.96046448e-08
   1.19209290e-07   1.49011612e-07   8.94069672e-08   1.78813934e-07
   4.17232513e-07   4.47034836e-07   4.47034836e-07   8.42846930e-08
  -3.70637281e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -1.04308128e-07   0.00000000e+00   3.35276127e-07   2.68220901e-07
   7.74860382e-07   7.15255737e-07   2.38418579e-07   1.19209290e-07
  -2.38418579e-07  -3.57627869e-07  -3.57627869e-07  -8.94069672e-08
   3.87430191e-07   3.57627869e-07   3.57627869e-07   5.96046448e-08
   2.98023224e-07   4.76837158e-07   5.36441803e-07   4.84287739e-08
   4.59044713e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -9.77270531e-09
  -1.19209290e-07   7.45058060e-09   0.00000000e+00   3.57627869e-07
   2.38418579e-07   5.96046448e-07   3.57627869e-07   9.53674316e-07
   4.76837158e-07   1.19209290e-07  -2.38418579e-07   1.19209290e-07
  -2.98023224e-07  -1.19209290e-07  -5.96046448e-08  -3.57627869e-07
   1.78813934e-07   4.47034836e-07   3.87430191e-07   1.08033419e-07
   3.81596692e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -4.18361026e-08
  -5.63450158e-08  -4.84287739e-08   5.96046448e-08   2.38418579e-07
   4.76837158e-07   3.57627869e-07   5.96046448e-07   1.78813934e-07
  -1.19209290e-07  -1.19209290e-07  -2.68220901e-07   0.00000000e+00
   0.00000000e+00   2.38418579e-07  -5.96046448e-07  -3.87430191e-07
   1.19209290e-07   3.57627869e-07   3.87430191e-07   1.24797225e-07
   6.91925379e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -4.36812968e-08
  -5.60366971e-08  -8.84756446e-08   1.19209290e-07  -1.19209290e-07
   3.57627869e-07   5.96046448e-08   0.00000000e+00   0.00000000e+00
  -5.96046448e-07   1.19209290e-07  -2.98023224e-08   1.78813934e-07
  -2.38418579e-07  -2.38418579e-07   3.57627869e-07  -3.57627869e-07
   2.38418579e-07   2.38418579e-07   3.42726707e-07   1.14087015e-07
   2.49773286e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -1.28651578e-09
  -1.01268569e-07  -1.50499545e-07  -4.47034836e-08   0.00000000e+00
   5.96046448e-08   3.57627869e-07   1.19209290e-07  -3.57627869e-07
  -3.57627869e-07   2.98023224e-07   3.12924385e-07   5.96046448e-08
   0.00000000e+00  -3.57627869e-07  -4.76837158e-07   5.96046448e-08
   2.98023224e-07   2.98023224e-07   1.78813934e-07   7.29633030e-08
   1.60424936e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -7.72104300e-08  -3.55039447e-08   2.83529516e-07   2.63098627e-08
   3.25031579e-07   7.45058060e-09   1.19209290e-07  -5.96046448e-08
  -3.27825546e-07   6.70552254e-08   3.12924385e-07   4.17232513e-07
  -3.57627869e-07  -4.76837158e-07  -1.19209290e-07   1.78813934e-07
   2.98023224e-07   1.71363354e-07   6.27478585e-08   1.92048901e-08
  -3.83870313e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -6.35419752e-08  -5.20981303e-08   2.64481059e-08   6.12053555e-08
   2.30735168e-07  -2.51457095e-08  -2.98023224e-07  -3.57627869e-07
   1.78813934e-07   6.55651093e-07   2.38418579e-07  -5.96046448e-08
  -3.57627869e-07   3.57627869e-07   1.19209290e-07   1.78813934e-07
   1.49011612e-07   8.19563866e-08   4.22005542e-08   2.06365929e-08
   2.06365929e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   2.09386428e-08   5.31318420e-08  -5.19480636e-09  -1.55114321e-07
   7.33416528e-08  -2.98023224e-08  -2.98023224e-08  -2.98023224e-07
   4.76837158e-07   2.38418579e-07   5.96046448e-08   3.57627869e-07
   1.19209290e-07   0.00000000e+00   2.38418579e-07   1.04308128e-07
   3.91155481e-08   4.49217623e-08   2.17316245e-08   2.06365929e-08
   2.23254459e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   1.64938871e-08   6.39862421e-08   5.16827185e-08   1.46355887e-07
  -1.34110451e-07  -5.96046448e-08   2.38418579e-07   7.74860382e-07
   9.53674316e-07   1.19209290e-07   0.00000000e+00   3.57627869e-07
   2.38418579e-07   0.00000000e+00   1.19209290e-07   1.44354999e-08
   2.26427801e-08   1.19798682e-09  -4.99121011e-09  -4.99121011e-09
  -2.63172950e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   3.20321369e-08   5.63813956e-08   8.19563866e-08
   0.00000000e+00  -1.19209290e-07   1.19209290e-07   1.19209290e-07
   1.19209290e-07  -1.19209290e-07   0.00000000e+00   0.00000000e+00
   5.96046448e-08  -7.45058060e-09   4.09781933e-08   1.63381628e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -7.45058060e-08  -5.96046448e-08
  -1.78813934e-07   5.96046448e-08  -2.88709998e-08  -3.35276127e-08
   6.28060661e-08   2.98023224e-08   1.19209290e-07   5.96046448e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -1.19209290e-07  -1.19209290e-07
   0.00000000e+00  -1.49011612e-08  -2.67755240e-08  -1.49157131e-09
   7.06131686e-09   6.01576176e-08  -2.69501470e-08   1.49011612e-08
   0.00000000e+00   0.00000000e+00  -7.45058060e-09   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]
<NDArray 784 @cpu(0)>


请注意,前几十个像素的权重梯度始终为零。由于调试器仅在ndarray上显示前几个值,因此您看到的只是零。

前几十个像素的权重梯度很可能为零,因为这些像素不包含任何可用于对图像中的数字进行分类的信息。请注意,MNIST图像在图像中居中。边界上的像素仅为0。

希望能有所帮助。

关于machine-learning - MXNet-将标量乘以数组将得出零,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/49326470/

10-12 22:11