Dr. CAN学习笔记-Ch0-1矩阵的导数运算
1. 标量向量方程对向量求导，分母布局，分子布局
1.1 标量方程对向量的导数
- y y y 为 一元向量 或 二元向量
- y y y为多元向量
y ⃗ = [ y 1 , y 2 , ⋯ , y n ] ⇒ ∂ f ( y ⃗ ) ∂ y ⃗ \vec{y}=\left[ y_1,y_2,\cdots ,y_{\mathrm{n}} \right] \Rightarrow \frac{\partial f\left( \vec{y} \right)}{\partial \vec{y}} y =[y1,y2,⋯,yn]⇒∂y ∂f(y )
其中： f ( y ⃗ ) f\left( \vec{y} \right) f(y ) 为标量 1 × 1 1\times 1 1×1, y ⃗ \vec{y} y 为向量 1 × n 1\times n 1×n
分母布局 Denominator Layout——行数与分母相同
∂ f ( y ⃗ ) ∂ y ⃗ = [ ∂ f ( y ⃗ ) ∂ y 1 ⋮ ∂ f ( y ⃗ ) ∂ y n ] n × 1 \frac{\partial f\left( \vec{y} \right)}{\partial \vec{y}}=\left[ \begin{array}{c} \frac{\partial f\left( \vec{y} \right)}{\partial y_1}\\ \vdots\\ \frac{\partial f\left( \vec{y} \right)}{\partial y_{\mathrm{n}}}\\ \end{array} \right] _{n\times 1} ∂y ∂f(y )= ∂y1∂f(y )⋮∂yn∂f(y ) n×1分子布局 Nunerator Layout——行数与分子相同
∂ f ( y ⃗ ) ∂ y ⃗ = [ ∂ f ( y ⃗ ) ∂ y 1 ⋯ ∂ f ( y ⃗ ) ∂ y n ] 1 × n \frac{\partial f\left( \vec{y} \right)}{\partial \vec{y}}=\left[ \begin{matrix} \frac{\partial f\left( \vec{y} \right)}{\partial y_1}& \cdots& \frac{\partial f\left( \vec{y} \right)}{\partial y_{\mathrm{n}}}\\ \end{matrix} \right] _{1\times n} ∂y ∂f(y )=[∂y1∂f(y )⋯∂yn∂f(y )]1×n
1.2 向量方程对向量的导数
f ⃗ ( y ⃗ ) = [ f ⃗ 1 ( y ⃗ ) ⋮ f ⃗ n ( y ⃗ ) ] n × 1 , y ⃗ = [ y 1 ⋮ y m ] m × 1 \vec{f}\left( \vec{y} \right) =\left[ \begin{array}{c} \vec{f}_1\left( \vec{y} \right)\\ \vdots\\ \vec{f}_{\mathrm{n}}\left( \vec{y} \right)\\ \end{array} \right] _{n\times 1},\vec{y}=\left[ \begin{array}{c} y_1\\ \vdots\\ y_{\mathrm{m}}\\ \end{array} \right] _{\mathrm{m}\times 1} f (y )= f 1(y )⋮f n(y ) n×1,y = y1⋮ym m×1
∂ f ⃗ ( y ⃗ ) n × 1 ∂ y ⃗ m × 1 = [ ∂ f ⃗ ( y ⃗ ) ∂ y 1 ⋮ ∂ f ⃗ ( y ⃗ ) ∂ y m ] m × 1 = [ ∂ f 1 ( y ⃗ ) ∂ y 1 ⋯ ∂ f n ( y ⃗ ) ∂ y 1 ⋮ ⋱ ⋮ ∂ f 1 ( y ⃗ ) ∂ y m ⋯ ∂ f n ( y ⃗ ) ∂ y m ] m × n \frac{\partial \vec{f}\left( \vec{y} \right) _{n\times 1}}{\partial \vec{y}_{\mathrm{m}\times 1}}=\left[ \begin{array}{c} \frac{\partial \vec{f}\left( \vec{y} \right)}{\partial y_1}\\ \vdots\\ \frac{\partial \vec{f}\left( \vec{y} \right)}{\partial y_{\mathrm{m}}}\\ \end{array} \right] _{\mathrm{m}\times 1}=\left[ \begin{matrix} \frac{\partial f_1\left( \vec{y} \right)}{\partial y_1}& \cdots& \frac{\partial f_{\mathrm{n}}\left( \vec{y} \right)}{\partial y_1}\\ \vdots& \ddots& \vdots\\ \frac{\partial f_1\left( \vec{y} \right)}{\partial y_{\mathrm{m}}}& \cdots& \frac{\partial f_{\mathrm{n}}\left( \vec{y} \right)}{\partial y_{\mathrm{m}}}\\ \end{matrix} \right] _{\mathrm{m}\times \mathrm{n}} ∂y m×1∂f (y )n×1= ∂y1∂f (y )⋮∂ym∂f (y ) m×1= ∂y1∂f1(y )⋮∂ym∂f1(y )⋯⋱⋯∂y1∂fn(y )⋮∂ym∂fn(y ) m×n, 为分母布局
若： y ⃗ = [ y 1 ⋮ y m ] m × 1 , A = [ a 11 ⋯ a 1 n ⋮ ⋱ ⋮ a m 1 ⋯ a m n ] \vec{y}=\left[ \begin{array}{c} y_1\\ \vdots\\ y_{\mathrm{m}}\\ \end{array} \right] _{\mathrm{m}\times 1}, A=\left[ \begin{matrix} a_{11}& \cdots& a_{1\mathrm{n}}\\ \vdots& \ddots& \vdots\\ a_{\mathrm{m}1}& \cdots& a_{\mathrm{mn}}\\ \end{matrix} \right] y = y1⋮ym m×1,A= a11⋮am1⋯⋱⋯a1n⋮amn , 则有：
- ∂ A y ⃗ ∂ y ⃗ = A T \frac{\partial A\vec{y}}{\partial \vec{y}}=A^{\mathrm{T}} ∂y ∂Ay =AT(分母布局)
- ∂ y ⃗ T A y ⃗ ∂ y ⃗ = A y ⃗ + A T y ⃗ \frac{\partial \vec{y}^{\mathrm{T}}A\vec{y}}{\partial \vec{y}}=A\vec{y}+A^{\mathrm{T}}\vec{y} ∂y ∂y TAy =Ay +ATy , 当 A = A T A=A^{\mathrm{T}} A=AT时, ∂ y ⃗ T A y ⃗ ∂ y ⃗ = 2 A y ⃗ \frac{\partial \vec{y}^{\mathrm{T}}A\vec{y}}{\partial \vec{y}}=2A\vec{y} ∂y ∂y TAy =2Ay
2. 案例分析，线性回归
Linear Regression 线性回归
z ^ = y 1 + y 2 x ⇒ J = ∑ i = 1 n [ z i − ( y 1 + y 2 x i ) ] 2 \hat{z}=y_1+y_2x\Rightarrow J=\sum_{i=1}^n{\left[ z_i-\left( y_1+y_2x_i \right) \right] ^2} z^=y1+y2x⇒J=i=1∑n[zi−(y1+y2xi)]2
找到 y 1 , y 2 y_1,y_2 y1,y2 使得 J J J最小
z ⃗ = [ z 1 ⋮ z n ] , [ x ⃗ ] = [ 1 x 1 ⋮ ⋮ 1 x n ] , y ⃗ = [ y 1 y 2 ] ⇒ z ⃗ ^ = [ x ⃗ ] y ⃗ = [ y 1 + y 2 x 1 ⋮ y 1 + y 2 x n ] \vec{z}=\left[ \begin{array}{c} z_1\\ \vdots\\ z_{\mathrm{n}}\\ \end{array} \right] ,\left[ \vec{x} \right] =\left[ \begin{array}{l} 1& x_1\\ \vdots& \vdots\\ 1& x_{\mathrm{n}}\\ \end{array} \right] ,\vec{y}=\left[ \begin{array}{c} y_1\\ y_2\\ \end{array} \right] \Rightarrow \hat{\vec{z}}=\left[ \vec{x} \right] \vec{y}=\left[ \begin{array}{c} y_1+y_2x_1\\ \vdots\\ y_1+y_2x_{\mathrm{n}}\\ \end{array} \right] z = z1⋮zn ,[x ]= 1⋮1x1⋮xn ,y =[y1y2]⇒z ^=[x ]y = y1+y2x1⋮y1+y2xn
J = [ z ⃗ − z ⃗ ^ ] T [ z ⃗ − z ⃗ ^ ] = [ z ⃗ − [ x ⃗ ] y ⃗ ] T [ z ⃗ − [ x ⃗ ] y ⃗ ] = z ⃗ z ⃗ T − z ⃗ T [ x ⃗ ] y ⃗ − y ⃗ T [ x ⃗ ] T z ⃗ + y ⃗ T [ x ⃗ ] T [ x ⃗ ] y ⃗ J=\left[ \vec{z}-\hat{\vec{z}} \right] ^{\mathrm{T}}\left[ \vec{z}-\hat{\vec{z}} \right] =\left[ \vec{z}-\left[ \vec{x} \right] \vec{y} \right] ^{\mathrm{T}}\left[ \vec{z}-\left[ \vec{x} \right] \vec{y} \right] =\vec{z}\vec{z}^{\mathrm{T}}-\vec{z}^{\mathrm{T}}\left[ \vec{x} \right] \vec{y}-\vec{y}^{\mathrm{T}}\left[ \vec{x} \right] ^{\mathrm{T}}\vec{z}+\vec{y}^{\mathrm{T}}\left[ \vec{x} \right] ^{\mathrm{T}}\left[ \vec{x} \right] \vec{y} J=[z −z ^]T[z −z ^]=[z −[x ]y ]T[z −[x ]y ]=z z T−z T[x ]y −y T[x ]Tz +y T[x ]T[x ]y
其中： ( z ⃗ T [ x ⃗ ] y ⃗ ) T = y ⃗ T [ x ⃗ ] T z ⃗ \left( \vec{z}^{\mathrm{T}}\left[ \vec{x} \right] \vec{y} \right) ^{\mathrm{T}}=\vec{y}^{\mathrm{T}}\left[ \vec{x} \right] ^{\mathrm{T}}\vec{z} (z T[x ]y )T=y T[x ]Tz ， 则有：
J = z ⃗ z ⃗ T − 2 z ⃗ T [ x ⃗ ] y ⃗ + y ⃗ T [ x ⃗ ] T [ x ⃗ ] y ⃗ J=\vec{z}\vec{z}^{\mathrm{T}}-2\vec{z}^{\mathrm{T}}\left[ \vec{x} \right] \vec{y}+\vec{y}^{\mathrm{T}}\left[ \vec{x} \right] ^{\mathrm{T}}\left[ \vec{x} \right] \vec{y} J=z z T−2z T[x ]y +y T[x ]T[x ]y
进而：
∂ J ∂ y ⃗ = 0 − 2 ( z ⃗ T [ x ⃗ ] ) T + 2 [ x ⃗ ] T [ x ⃗ ] y ⃗ = ∇ y ⃗ ⟹ ∂ J ∂ y ⃗ ∗ = 0 , y ⃗ ∗ = ( [ x ⃗ ] T [ x ⃗ ] ) − 1 [ x ⃗ ] T z ⃗ \frac{\partial J}{\partial \vec{y}}=0-2\left( \vec{z}^{\mathrm{T}}\left[ \vec{x} \right] \right) ^{\mathrm{T}}+2\left[ \vec{x} \right] ^{\mathrm{T}}\left[ \vec{x} \right] \vec{y}=\nabla \vec{y}\Longrightarrow \frac{\partial J}{\partial \vec{y}^*}=0,\vec{y}^*=\left( \left[ \vec{x} \right] ^{\mathrm{T}}\left[ \vec{x} \right] \right) ^{-1}\left[ \vec{x} \right] ^{\mathrm{T}}\vec{z} ∂y ∂J=0−2(z T[x ])T+2[x ]T[x ]y =∇y ⟹∂y ∗∂J=0,y ∗=([x ]T[x ])−1[x ]Tz
其中： ( [ x ⃗ ] T [ x ⃗ ] ) − 1 \left( \left[ \vec{x} \right] ^{\mathrm{T}}\left[ \vec{x} \right] \right) ^{-1} ([x ]T[x ])−1不一定有解，则 y ⃗ ∗ \vec{y}^* y ∗无法得到解析解——定义初始 y ⃗ ∗ \vec{y}^* y ∗， y ⃗ ∗ = y ⃗ ∗ − α ∇ , α = [ α 1 0 0 α 2 ] \vec{y}^*=\vec{y}^*-\alpha \nabla ,\alpha =\left[ \begin{matrix} \alpha _1& 0\\ 0& \alpha _2\\ \end{matrix} \right] y ∗=y ∗−α∇,α=[α100α2]
其中： α \alpha α称为学习率，对 x x x而言则需进行归一化
3. 矩阵求导的链式法则
标量函数： J = f ( y ( u ) ) , ∂ J ∂ u = ∂ J ∂ y ∂ y ∂ u J=f\left( y\left( u \right) \right) ,\frac{\partial J}{\partial u}=\frac{\partial J}{\partial y}\frac{\partial y}{\partial u} J=f(y(u)),∂u∂J=∂y∂J∂u∂y
标量对向量求导： J = f ( y ⃗ ( u ⃗ ) ) , y ⃗ = [ y 1 ( u ⃗ ) ⋮ y m ( u ⃗ ) ] m × 1 , u ⃗ = [ u ⃗ 1 ⋮ u ⃗ n ] n × 1 J=f\left( \vec{y}\left( \vec{u} \right) \right) ,\vec{y}=\left[ \begin{array}{c} y_1\left( \vec{u} \right)\\ \vdots\\ y_{\mathrm{m}}\left( \vec{u} \right)\\ \end{array} \right] _{m\times 1},\vec{u}=\left[ \begin{array}{c} \vec{u}_1\\ \vdots\\ \vec{u}_{\mathrm{n}}\\ \end{array} \right] _{\mathrm{n}\times 1} J=f(y (u )),y = y1(u )⋮ym(u ) m×1,u = u 1⋮u n n×1
分析： ∂ J 1 × 1 ∂ u n × 1 n × 1 = ∂ J ∂ y m × 1 m × 1 ∂ y m × 1 ∂ u n × 1 n × m \frac{\partial J_{1\times 1}}{\partial u_{\mathrm{n}\times 1}}_{\mathrm{n}\times 1}=\frac{\partial J}{\partial y_{m\times 1}}_{m\times 1}\frac{\partial y_{m\times 1}}{\partial u_{\mathrm{n}\times 1}}_{\mathrm{n}\times \mathrm{m}} ∂un×1∂J1×1n×1=∂ym×1∂Jm×1∂un×1∂ym×1n×m 无法相乘
∂ J ∂ u ⃗ = ∂ y ⃗ ( u ⃗ ) ∂ u ⃗ ∂ J ∂ y ⃗ \frac{\partial J}{\partial \vec{u}}=\frac{\partial \vec{y}\left( \vec{u} \right)}{\partial \vec{u}}\frac{\partial J}{\partial \vec{y}} ∂u ∂J=∂u ∂y (u )∂y ∂J
eg:
x ⃗ [ k + 1 ] = A x ⃗ [ k ] + B u ⃗ [ k ] , J = x ⃗ T [ k + 1 ] x ⃗ [ k + 1 ] \vec{x}\left[ k+1 \right] =A\vec{x}\left[ k \right] +B\vec{u}\left[ k \right] ,J=\vec{x}^{\mathrm{T}}\left[ k+1 \right] \vec{x}\left[ k+1 \right] x [k+1]=Ax [k]+Bu [k],J=x T[k+1]x [k+1]
∂ J ∂ u ⃗ = ∂ x ⃗ [ k + 1 ] ∂ u ⃗ ∂ J ∂ x ⃗ [ k + 1 ] = B T ⋅ 2 x ⃗ [ k + 1 ] = 2 B T x ⃗ [ k + 1 ] \frac{\partial J}{\partial \vec{u}}=\frac{\partial \vec{x}\left[ k+1 \right]}{\partial \vec{u}}\frac{\partial J}{\partial \vec{x}\left[ k+1 \right]}=B^{\mathrm{T}}\cdot 2\vec{x}\left[ k+1 \right] =2B^{\mathrm{T}}\vec{x}\left[ k+1 \right] ∂u ∂J=∂u ∂x [k+1]∂x [k+1]∂J=BT⋅2x [k+1]=2BTx [k+1]