神经网络推导-批量数据

输入批量数据到神经网络，进行前向传播和反向传播的推导

TestNet网络

TestNet是一个2层神经网络，结构如下：

• 输入层有3个神经元
• 隐藏层有4个神经元
• 输出层有2个神经元

• 激活函数为relu函数
• 评分函数为softmax回归
• 代价函数为交叉熵损失

网络符号定义

规范神经网络的计算符号

关于神经元和层数

• $L$表示网络层数（不计入输入层）
  • $L=2$，其中输入层是第0层，隐藏层是第1层，输出层是第2层
• $n^{(l)}$表示第$l$层的神经元个数（不包括偏置神经元）
  • $n^{(0)}=3$，表示输入层神经元个数为3
  • $n^{(1)}=4$，表示隐藏层神经元个数为4
  • $n^{(2)}=2$，表示输出层神经元个数为2

关于权重矩阵和偏置值

• $W^{(l)}$表示第$l-1$层到第$l$层的权重矩阵，矩阵行数为第$l-1$层的神经元个数，列数为第$l$层神经元个数
  • $W^{(1)}$表示输入层到隐藏层的权重矩阵，大小为$R^{3\times 4}$
  • $W^{(2)}$表示隐藏层到输出层的权重矩阵，大小为$R^{4\times 2}$
• $W_{i,j}^{(l)}$表示第$l-1$层第$i$个神经元到第$l$第$j$个神经元的权值
  • $i$的取值范围是$[1,n^{(l-1)}]$
  • $j$的取值范围是$[1,n^{(l)}]$
• $W_i^{(l)}$表示第$l-1$层第$i$个神经元对应的权重向量，大小为$n^{(l)}$
• $W_{,j}^{(l)}$表示第$l$层第$j$个神经元对应的权重向量，大小为$n^{(l-1)}$
• $b^{(l)}$表示第$l$层的偏置向量
  • $b^{(1)}$表示输入层到隐藏层的偏置向量，大小为$R^{1\times 4}$
  • $b^{(2)}$表示隐藏层到输出层的偏置向量，大小为$R^{1\times 2}$
• $b_i^{(l)}$表示第$l$层第$i$个神经元的偏置值
  • $b_2^{(1)}$表示第$1$层隐藏层第$2$个神经元的偏置值

关于神经元输入向量和输出向量

• $a^{(l)}$表示第$l$层输出向量，$a^{(l)}=[a_1^{(l)},a_2^{(l)},...,a_m^{(l)}{]}^T$
  • $a^{(0)}$表示输入层输出向量，大小为$R^{m\times 3}$
  • $a^{(1)}$表示隐藏层输出向量，大小为$R^{m\times 4}$
  • $a^{(2)}$表示输出层输出向量，大小为$R^{m\times 2}$
• $a_i^{(l)}$表示第$l$层第$i$个单元的输出值，其是输入向量经过激活计算后的值
  • $a_3^{(1)}$表示隐含层第$3$个神经元的输入值，$a_3^{(1)}=g(z_3^{(1)})$
• $z^{(l)}$表示第$l$层输入向量，$z^{(l)}=[z_1^{(l)},z_2^{(l)},...,z_m^{(l)}{]}^T$
  • $z^{(1)}$表示隐藏层的输入向量，大小为$R^{m\times 4}$
  • $z^{(2)}$表示输出层的输入向量，大小为$R^{m\times 2}$
• $z_{i,j}^{(l)}$表示第$l$层第$j$个单元的输入值，其是上一层输出向量第$i$个数据和该层第$j$个神经元权重向量的加权累加和
  • $z_{1,2}^{(1)}$表示隐藏层第$2$个神经元的输入值，$z_{1,2}^{(1)}=b_2^{(2)}+a_{1,1}^{(0)}\cdot W_{1,2}^{(1)}+a_{1,2}^{(0)}\cdot W_{2,2}^{(1)}+a_{1,3}^{(0)}\cdot W_{3,2}^{(1)}$

关于神经元激活函数

• $g()$表示激活函数操作

关于评分函数和损失函数

• $h()$表示评分函数操作
• $J()$表示代价函数操作

神经元执行步骤

神经元操作分为2步计算：

1. 输入向量$z^{(l)}$=前一层神经元输出向量$a^{(l-1)}$与权重矩阵$W^{(l)}$的加权累加和+偏置向量

$$z_{i,j}^{(l)}=a_i^{(l-1)}\cdot W_{,j}^{(l)}+b_j^{(l)}\Rightarrow z^{(l)}=a^{(l-1)}\cdot W^{(l)}+b^{(l)}$$

2. 输出向量$a^{(l)}$=对输入向量$z^{(l)}$进行激活函数操作

$$a_i^{(l)}=g(z_i^{(l)})\Rightarrow a^{(l)}=g(z^{(l)})$$

网络结构

对输入层

$$a^{(0)}=\left[\begin{array}{c}{a}_1^{(0)}\\ ⋮\\ a_m^{(0)}\end{array}\right]=\left[\begin{array}{ccc}{a}_{1,1}^{(0)}& a_{1,2}^{(0)}& a_{1,3}^{(0)}\\ ⋮& ⋮& ⋮\\ a_{m,1}^{(0)}& a_{m,2}^{(0)}& a_{m,3}^{(0)}\end{array}\right]\in R^{m\times 3}$$

对隐藏层

$$W^{(1)}=\left[\begin{array}{cccc}{W}_{1,1}^{(1)}& W_{1,2}^{(1)}& W_{1,3}^{(1)}& W_{1,4}^{(1)}\\ W_{2,1}^{(1)}& W_{2,2}^{(1)}& W_{2,3}^{(1)}& W_{2,4}^{(1)}\\ W_{3,1}^{(1)}& W_{3,2}^{(1)}& W_{3,3}^{(1)}& W_{3,4}^{(1)}\end{array}\right]\in R^{3\times 4}$$

$$b^{(1)}=[[b_1^{(1)},b_2^{(1)},b_3^{(1)},b_4^{(1)}]]\in R^{1\times 4}$$

$$z^{(1)}=\left[\begin{array}{cccc}{z}_{1,1}^{(0)}& z_{1,2}^{(0)}& z_{1,3}^{(0)}& z_{1,4}^{(0)}\\ ⋮& ⋮& ⋮& ⋮\\ z_{m,1}^{(0)}& z_{m,2}^{(0)}& z_{m,3}^{(0)}& z_{m,4}^{(0)}\end{array}\right]\in R^{m\times 4}$$

$$a^{(1)}=\left[\begin{array}{cccc}{a}_{1,1}^{(0)}& a_{1,2}^{(0)}& a_{1,3}^{(0)}& a_{1,4}^{(0)}\\ ⋮& ⋮& ⋮& ⋮\\ a_{m,1}^{(0)}& a_{m,2}^{(0)}& a_{m,3}^{(0)}& a_{m,4}^{(0)}\end{array}\right]\in R^{m\times 4}$$

对输出层

$$W^{(2)}=\left[\begin{array}{cc}{W}_{1,1}^{(2)}& W_{1,2}^{(2)}\\ W_{2,1}^{(2)}& W_{2,2}^{(2)}\\ W_{3,1}^{(2)}& W_{3,2}^{(2)}\\ W_{4,1}^{(2)}& W_{4,2}^{(2)}\end{array}\right]\in R^{4\times 2}$$

$$b^{(2)}=[[b_1^{(2)},b_2^{(2)}]]\in R^{1\times 2}$$

$$z^{(2)}=\left[\begin{array}{cc}{z}_{1,1}^{(2)}& z_{1,2}^{(0)}\\ ⋮& ⋮\\ z_{m,1}^{(2)}& z_{m,2}^{(0)}\end{array}\right]\in R^{m\times 2}$$

评分值

$$h(z^{(2)})=\left[\begin{array}{cc}p(y_1=1)& p(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)& p(y_m=2)\end{array}\right]\in R^{m\times 2}$$

损失值

$$J(z^{(2)})=(-1)\sum _{i=1}^m\sum _{j=1}^2\cdot 1(y_{m,j}=1)\ln p(y_{m,j}=1)\in R^1$$

前向传播

输入层到隐藏层计算

$$z_{i,1}^{(1)}=a_i^{(0)}\cdot W_{,1}^{(1)}+b_1^{(1)}=a_{i,1}^{(0)}\cdot W_{1,1}^{(1)}+a_{i,2}^{(0)}\cdot W_{2,1}^{(1)}+a_{i,3}^{(0)}\cdot W_{3,1}^{(1)}+b_{1,1}^{(1)}$$

$$z_{i,2}^{(1)}=a_i^{(0)}\cdot W_{,2}^{(1)}+b_2^{(1)}=a_{i,1}^{(0)}\cdot W_{1,2}^{(1)}+a_{i,2}^{(0)}\cdot W_{2,2}^{(1)}+a_{i,3}^{(0)}\cdot W_{3,2}^{(1)}+b_{1,2}^{(1)}$$

$$z_{i,3}^{(1)}=a_i^{(0)}\cdot W_{,3}^{(1)}+b_3^{(1)}=a_{i,1}^{(0)}\cdot W_{1,3}^{(1)}+a_{i,2}^{(0)}\cdot W_{2,3}^{(1)}+a_{i,3}^{(0)}\cdot W_{3,3}^{(1)}+b_{1,3}^{(1)}$$

$$z_{i,4}^{(1)}=a_i^{(0)}\cdot W_{,4}^{(1)}+b_4^{(1)}=a_{i,1}^{(0)}\cdot W_{1,4}^{(1)}+a_{i,2}^{(0)}\cdot W_{2,4}^{(1)}+a_{i,3}^{(0)}\cdot W_{3,4}^{(1)}+b_{1,4}^{(1)}$$

$$\Rightarrow z_i^{(1)}=[z_{i,1}^{(1)},z_{i,2}^{(1)},z_{i,3}^{(1)},z_{i,4}^{(1)}]=a_i^{(0)}\cdot W^{(1)}+b^{(1)}$$

$$\Rightarrow z^{(1)}=a^{(0)}\cdot W^{(1)}+b^{(1)}$$

隐藏层输入向量到输出向量

$$\begin{gathered} a_{i,1}^{(1)}=relu(z_{i,1}^{(1)}) \\ {a}_{i,2}^{(1)}=relu(z_{i,2}^{(1)}) \\ {a}_{i,3}^{(1)}=relu(z_{i,3}^{(1)}) \\ {a}_{i,4}^{(1)}=relu(z_{i,4}^{(1)}) \end{gathered}$$

$$\Rightarrow a_i^{(1)}=[a_{i,1}^{(1)},a_{i,2}^{(1)},a_{i,3}^{(1)},a_{i,4}^{(1)}]=relu(z_i^{(1)})$$

$$\Rightarrow a^{(1)}=relu(z^{(1)})$$

隐藏层到输出层计算

$$z_{i,1}^{(2)}=a_i^{(1)}\cdot W_{,1}^{(2)}+b_{1,1}^{(2)}=a_{i,1}^{(1)}\cdot W_{1,1}^{(2)}+a_{i,2}^{(1)}\cdot W_{2,1}^{(2)}+a_{i,3}^{(1)}\cdot W_{3,1}^{(2)}+a_{i,4}^{(1)}\cdot W_{4,1}^{(2)}+b_{1,1}^{(2)}$$

$$z_{i,2}^{(2)}=a_i^{(1)}\cdot W_{,2}^{(2)}+b_{1,2}^{(2)}=a_{i,1}^{(1)}\cdot W_{1,2}^{(2)}+a_{i,2}^{(1)}\cdot W_{2,2}^{(2)}+a_{i,3}^{(1)}\cdot W_{3,2}^{(2)}+a_{i,4}^{(1)}\cdot W_{4,2}^{(2)}+b_{1,2}^{(2)}$$

$$\Rightarrow z_i^{(2)}=[z_{i,1}^{(2)},z_{i,2}^{(2)}]=a_i^{(1)}\cdot W^{(2)}+b^{(2)}$$

$$\Rightarrow z^{(2)}=a^{(1)}\cdot W^{(2)}+b^{(2)}$$

评分操作

$$\begin{gathered} p(y_i=1)=\frac{exp(z_{i,1}^{(2)})}{\sum exp(z_i^{(2)})} \\ p(y_i=2)=\frac{exp(z_{i,2}^{(2)})}{\sum exp(z_i^{(2)})} \end{gathered}$$

$$\Rightarrow h(z_i^{(2)})=[p(y_i=1),p(y_i=2)]=[\frac{exp(z_{i,1}^{(2)})}{\sum exp(z_i^{(2)})},\frac{exp(z_{i,2}^{(2)})}{\sum exp(z_i^{(2)})}]$$

$$\Rightarrow h(z^{(2)})=\left[\begin{array}{cc}p(y_1=1)& p(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)& p(y_m=2)\end{array}\right]$$

损失值

$$J(z^{(2)})=(-1)\sum _{i=1}^m\sum _{j=1}^2\cdot 1(y_{m,j}=1)\ln p(y_{m,j}=1)$$

反向传播

计算输出层输入向量梯度

$$\frac{\partial J}{\partial z_{i,1}^{(2)}}=(-1)\cdot \frac{1(y_i=1)}{p(y_i=1)}\cdot \frac{\partial p(y_i=1)}{\partial z_{i,1}^{(2)}}+(-1)\cdot \frac{1(y_i=2)}{p(y_i=2)}\cdot \frac{\partial p(y_i=2)}{\partial z_{i,1}^{(2)}}$$

$$\frac{\partial p(y_i=1)}{\partial z_{i,1}^{(2)}}=\frac{exp(z_{i,1}^{(2)})\cdot \sum exp(z_i^{(2)})-exp(z_{i,1}^{(2)})\cdot exp(z_{i,1}^{(2)})}{(\sum exp(z_i^{(2)}){)}^2}=\frac{exp(z_{i,1}^{(2)})}{\sum exp(z_i^{(2)})}-(\frac{exp(z_{i,1}^{(2)})}{\sum exp(z_i^{(2)})}{)}^2=p(y_i=1)-(p(y_i=1){)}^2$$

$$\frac{\partial p(y_i=2)}{\partial z_{i,1}^{(2)}}=\frac{-exp(z_{i,2}^{(2)})\cdot exp(z_{i,1}^{(2)})}{(\sum exp(z_i^{(2)}){)}^2}=(-1)\cdot \frac{exp(z_{i,1}^{(2)})}{\sum exp(z_i^{(2)})}\cdot \frac{exp(z_{i,2}^{(2)})}{\sum exp(z_i^{(2)})}=(-1)\cdot p(y_i=1)p(y_i=2)$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial z_{i,1}^{(2)}}=(-1)\cdot \frac{1(y_i=1)}{p(y_i=1)}\cdot (p(y_i=1)-(p(y_i=1){)}^2)+(-1)\cdot \frac{1(y_i=2)}{p(y_i=2)}\cdot (-1)\cdot p(y_i=1)p(y_i=2) \\ =(-1)\cdot 1(y_i=1)\cdot (1-p(y_i=1))+1(y_i=2)\cdot p(y_i=1)=p(y_i=1)-1(y_i=1) \end{gathered}$$

$$\Rightarrow \frac{\partial J}{\partial z_{i,2}^{(2)}}=p(y_i=2)-1(y_i=2)$$

$$\Rightarrow \frac{\partial J}{\partial z_i^{(2)}}=[p(y_i=1)-1(y_i=1),p(y_i=2)-1(y_i=2)]$$

$$\Rightarrow \frac{\partial J}{\partial z^{(2)}}=\left[\begin{array}{cc}p(y_1=1)-1(y_1=1)& p(y_1=2)-1(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)-1(y_m=1)& p(y_m=2)-1(y_m=2)\end{array}\right]$$

计算输出层权重向量梯度

$$\frac{\partial J}{\partial W_{1,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial W_{1,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,1}^{(1)})$$

$$\frac{\partial J}{\partial W_{2,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial W_{2,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,2}^{(1)})$$

$$\frac{\partial J}{\partial W_{3,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial W_{3,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,3}^{(1)})$$

$$\frac{\partial J}{\partial W_{4,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial W_{4,1}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,4}^{(1)})$$

$$\frac{\partial J}{\partial W_{1,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_2^{(2)}}{\partial W_{1,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,1}^{(1)})$$

$$\frac{\partial J}{\partial W_{2,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_2^{(2)}}{\partial W_{2,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,2}^{(1)})$$

$$\frac{\partial J}{\partial W_{3,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_2^{(2)}}{\partial W_{3,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,3}^{(1)})$$

$$\frac{\partial J}{\partial W_{4,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_2^{(2)}}{\partial W_{4,2}^{(2)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,4}^{(1)})$$

$$\Rightarrow \frac{\partial J}{\partial W^{(2)}}=\left[\begin{array}{cc}\frac{\partial J}{\partial W_{1,1}^{(2)}}& \frac{\partial J}{\partial W_{1,2}^{(2)}}\\ \frac{\partial J}{\partial W_{2,1}^{(2)}}& \frac{\partial J}{\partial W_{2,2}^{(2)}}\\ \frac{\partial J}{\partial W_{3,1}^{(2)}}& \frac{\partial J}{\partial W_{3,2}^{(2)}}\\ \frac{\partial J}{\partial W_{4,1}^{(2)}}& \frac{\partial J}{\partial W_{4,2}^{(2)}}\end{array}\right]$$

$$=\left[\begin{array}{cc}\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,1}^{(1)})& \frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,2}^{(1)})\\ \frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,3}^{(1)})& \frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot a_{i,4}^{(1)})\\ \frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,1}^{(1)})& \frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,2}^{(1)})\\ \frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,3}^{(1)})& \frac{1}{m}\sum _{i=1}^m((p(y_i=2)-1(y_i=2))\cdot a_{i,4}^{(1)})\end{array}\right]$$

$$\begin{gathered} =\frac{1}{m}\sum _{i=1}^m\left[\begin{array}{c}{a}_{i,1}^{(1)}\\ a_{i,2}^{(1)}\\ a_{i,3}^{(1)}\\ a_{i,4}^{(1)}\end{array}\right]\left[\begin{array}{cc}p(y_i=1)-1(y_i=1)& p(y_i=2)-1(y_i=2)\end{array}\right] \\ =\frac{1}{m}\sum _{i=1}^m((a_i^{(1)}{)}^T\cdot \frac{\partial J}{\partial z_i^{(2)}})=\frac{1}{m}(a^{(1)}{)}^T\cdot \frac{\partial J}{\partial z^{(2)}}=\frac{1}{m}\sum _{i=1}^m(R^{4\times m}\cdot R^{m\times 2})=R^{4\times 2} \end{gathered}$$

计算隐藏层输出向量梯度

$$\frac{\partial J}{\partial a_{i,1}^{(1)}}=\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial a_{i,1}^{(1)}}+\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_{i,2}^{(2)}}{\partial a_{i,1}^{(1)}}=(p(y_i=1)-1(y_i=1))\cdot W_{1,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{1,2}^{(2)}$$

$$\frac{\partial J}{\partial a_{i,2}^{(1)}}=\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial a_{i,2}^{(1)}}+\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_{i,2}^{(2)}}{\partial a_{i,2}^{(1)}}=(p(y_i=1)-1(y_i=1))\cdot W_{2,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{2,2}^{(2)}$$

$$\frac{\partial J}{\partial a_{i,3}^{(1)}}=\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial a_{i,3}^{(1)}}+\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_{i,2}^{(2)}}{\partial a_{i,3}^{(1)}}=(p(y_i=1)-1(y_i=1))\cdot W_{3,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{3,2}^{(2)}$$

$$\frac{\partial J}{\partial a_{i,4}^{(1)}}=\frac{\partial J}{\partial z_{i,1}^{(2)}}\cdot \frac{\partial z_{i,1}^{(2)}}{\partial a_{i,4}^{(1)}}+\frac{\partial J}{\partial z_{i,2}^{(2)}}\cdot \frac{\partial z_{i,2}^{(2)}}{\partial a_{i,4}^{(1)}}=(p(y_i=1)-1(y_i=1))\cdot W_{4,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{4,2}^{(2)}$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial a_i^{(1)}}=\left[\begin{array}{cc}p(y_i=1)-1(y_i=1)& p(y_i=2)-1(y_i=2)\end{array}\right]\left[\begin{array}{cccc}{W}_{1,1}^{(2)}& W_{2,1}^{(2)}& W_{3,1}^{(2)}& W_{4,1}^{(2)}\\ W_{1,2}^{(2)}& W_{2,2}^{(2)}& W_{3,2}^{(2)}& W_{4,2}^{(2)}\end{array}\right] \\ =\frac{\partial J}{\partial z_i^{(2)}}\cdot (W^{(2)}{)}^T=R^{1\times 2}\cdot R^{2\times 4}=R^{1\times 4} \end{gathered}$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial a^{(1)}}=\left[\begin{array}{cc}p(y_1=1)-1(y_1=1)& p(y_1=2)-1(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)-1(y_m=1)& p(y_m=2)-1(y_m=2)\end{array}\right]\left[\begin{array}{cccc}{W}_{1,1}^{(2)}& W_{2,1}^{(2)}& W_{3,1}^{(2)}& W_{4,1}^{(2)}\\ W_{1,2}^{(2)}& W_{2,2}^{(2)}& W_{3,2}^{(2)}& W_{4,2}^{(2)}\end{array}\right] \\ =\frac{\partial J}{\partial z^{(2)}}\cdot (W^{(2)}{)}^T=R^{m\times 2}\cdot R^{2\times 4}=R^{m\times 4} \end{gathered}$$

计算隐藏层输入向量的梯度

$$\frac{\partial J}{\partial z_{i,1}^{(1)}}=\frac{\partial J}{\partial a_{i,1}^{(1)}}\cdot \frac{\partial a_{i,1}^{(1)}}{\partial z_{i,1}^{(1)}}=((p(y_i=1)-1(y_i=1))\cdot W_{1,1}^{(2)}+(p(y=2)-1(y=2))\cdot W_{1,2}^{(2)})\cdot 1(z_{i,1}^{(1)}\ge 0)$$

$$\frac{\partial J}{\partial z_{i,2}^{(1)}}=\frac{\partial J}{\partial a_{i,2}^{(1)}}\cdot \frac{\partial a_{i,2}^{(1)}}{\partial z_{i,2}^{(1)}}=((p(y_i=1)-1(y_i=1))\cdot W_{2,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{2,2}^{(2)})\cdot 1(z_{i,2}^{(1)}\ge 0)$$

$$\frac{\partial J}{\partial z_{i,3}^{(1)}}=\frac{\partial J}{\partial a_{i,3}^{(1)}}\cdot \frac{\partial a_{i,3}^{(1)}}{\partial z_{i,3}^{(1)}}=((p(y_i=1)-1(y_i=1))\cdot W_{3,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{3,2}^{(2)})\cdot 1(z_{i,3}^{(1)}\ge 0)$$

$$\frac{\partial J}{\partial z_{i,4}^{(1)}}=\frac{\partial J}{\partial a_{i,4}^{(1)}}\cdot \frac{\partial a_{i,4}^{(1)}}{\partial z_{i,4}^{(1)}}=((p(y_i=1)-1(y_i=1))\cdot W_{4,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{4,2}^{(2)})\cdot 1(z_{i,4}^{(1)}\ge 0)$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial z_i^{(1)}}=(\left[\begin{array}{cc}p(y_i=1)-1(y_i=1)& p(y_i=2)-1(y_i=2)\end{array}\right]\left[\begin{array}{cccc}{W}_{1,1}^{(2)}& W_{2,1}^{(2)}& W_{3,1}^{(2)}& W_{4,1}^{(2)}\\ W_{1,2}^{(2)}& W_{2,2}^{(2)}& W_{3,2}^{(2)}& W_{4,2}^{(2)}\end{array}\right])\ast \left[\begin{array}{cccc}\frac{\partial a_{i,1}^{(1)}}{\partial z_{i,1}^{(1)}}& \frac{\partial a_{i,2}^{(1)}}{\partial z_{i,2}^{(1)}}& \frac{\partial a_{i,3}^{(1)}}{\partial z_{i,3}^{(1)}}& \frac{\partial a_{i,4}^{(1)}}{\partial z_{i,4}^{(1)}}\end{array}\right] \\ =(R^{1\times 2}\cdot R^{2\times 4})\ast R^{1\times 4}=R^{1\times 4} \end{gathered}$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial z_i^{(1)}}=(\left[\begin{array}{cc}p(y_i=1)-1(y_i=1)& p(y_i=2)-1(y_i=2)\end{array}\right]\left[\begin{array}{cccc}{W}_{1,1}^{(2)}& W_{2,1}^{(2)}& W_{3,1}^{(2)}& W_{4,1}^{(2)}\\ W_{1,2}^{(2)}& W_{2,2}^{(2)}& W_{3,2}^{(2)}& W_{4,2}^{(2)}\end{array}\right])\ast \left[\begin{array}{cccc}1(z_{i,1}^{(1)}\ge 0)& 1(z_{i,2}^{(1)}\ge 0)& 1(z_{i,3}^{(1)}\ge 0)& 1(z_{i,4}^{(1)}\ge 0)\end{array}\right] \\ =(R^{1\times 2}\cdot R^{2\times 4})\ast R^{1\times 4}=R^{1\times 4} \end{gathered}$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial z^{(1)}}=(\left[\begin{array}{cc}p(y_1=1)-1(y_1=1)& p(y_1=2)-1(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)-1(y_m=1)& p(y_m=2)-1(y_m=2)\end{array}\right]\left[\begin{array}{cccc}{W}_{1,1}^{(2)}& W_{2,1}^{(2)}& W_{3,1}^{(2)}& W_{4,1}^{(2)}\\ W_{1,2}^{(2)}& W_{2,2}^{(2)}& W_{3,2}^{(2)}& W_{4,2}^{(2)}\end{array}\right])\ast \left[\begin{array}{cccc}1(z_{1,1}^{(1)}\ge 0)& 1(z_{1,2}^{(1)}\ge 0)& 1(z_{1,3}^{(1)}\ge 0)& 1(z_{1,4}^{(1)}\ge 0)\\ ⋮& ⋮\\ 1(z_{m,1}^{(1)}\ge 0)& 1(z_{m,2}^{(1)}\ge 0)& 1(z_{m,3}^{(1)}\ge 0)& 1(z_{m,4}^{(1)}\ge 0)\end{array}\right] \\ =\frac{\partial J}{\partial a^{(1)}}\ast 1(z^{(1)}\ge 0)=(R^{m\times 2}\cdot R^{2\times 4})\ast R^{m\times 4}=R^{m\times 4} \end{gathered}$$

计算隐藏层权重向量的梯度

$$\frac{\partial J}{\partial W_{1,1}^{(1)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{1,1}^{(1)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot W_{1,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{1,2}^{(2)})\cdot 1(z_{i,1}^{(1)}\ge 0)\cdot a_{i,1}^{(0)}$$

$$\frac{\partial J}{\partial W_{1,2}^{(1)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{1,2}^{(1)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot W_{2,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{2,2}^{(2)})\cdot 1(z_{i,2}^{(1)}\ge 0)\cdot a_{i,1}^{(0)}$$

$$\Rightarrow \frac{\partial J}{\partial W_{k,l}^{(1)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,l}^{(1)}}\cdot \frac{\partial z_{i,l}^{(1)}}{\partial W_{k,l}^{(1)}}=\frac{1}{m}\sum _{i=1}^m((p(y_i=1)-1(y_i=1))\cdot W_{l,1}^{(2)}+(p(y_i=2)-1(y_i=2))\cdot W_{l,2}^{(2)})\cdot 1(z_{i,l}^{(1)}\ge 0)\cdot a_{i,k}^{(0)}$$

$$\begin{gathered} \Rightarrow \frac{\partial J}{\partial W^{(1)}}=\left[\begin{array}{cccc}\frac{\partial J}{\partial W_{1,1}^{(1)}}& \frac{\partial J}{\partial W_{1,2}^{(1)}}& \frac{\partial J}{\partial W_{1,3}^{(1)}}& \frac{\partial J}{\partial W_{1,4}^{(1)}}\\ \frac{\partial J}{\partial W_{2,1}^{(1)}}& \frac{\partial J}{\partial W_{2,2}^{(1)}}& \frac{\partial J}{\partial W_{2,3}^{(1)}}& \frac{\partial J}{\partial W_{2,4}^{(1)}}\\ \frac{\partial J}{\partial W_{3,1}^{(1)}}& \frac{\partial J}{\partial W_{3,2}^{(1)}}& \frac{\partial J}{\partial W_{3,3}^{(1)}}& \frac{\partial J}{\partial W_{3,4}^{(1)}}\end{array}\right] \\ =\left[\begin{array}{cccc}\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{1,1}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{1,2}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot \frac{\partial z_{i,3}^{(1)}}{\partial W_{1,3}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot \frac{\partial z_{i,4}^{(1)}}{\partial W_{1,4}^{(1)}}\\ \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{2,1}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{2,2}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot \frac{\partial z_{i,3}^{(1)}}{\partial W_{2,3}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot \frac{\partial z_{i,4}^{(1)}}{\partial W_{2,4}^{(1)}}\\ \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{3,1}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{3,2}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot \frac{\partial z_{i,3}^{(1)}}{\partial W_{3,3}^{(1)}}& \frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot \frac{\partial z_{i,4}^{(1)}}{\partial W_{3,4}^{(1)}}\end{array}\right] \\ =\frac{1}{m}\sum _{i=1}^m\left[\begin{array}{cccc}\frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{1,1}^{(1)}}& \frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{1,2}^{(1)}}& \frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot \frac{\partial z_{i,3}^{(1)}}{\partial W_{1,3}^{(1)}}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot \frac{\partial z_{i,4}^{(1)}}{\partial W_{1,4}^{(1)}}\\ \frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{2,1}^{(1)}}& \frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{2,2}^{(1)}}& \frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot \frac{\partial z_{i,3}^{(1)}}{\partial W_{2,3}^{(1)}}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot \frac{\partial z_{i,4}^{(1)}}{\partial W_{2,4}^{(1)}}\\ \frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot \frac{\partial z_{i,1}^{(1)}}{\partial W_{3,1}^{(1)}}& \frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot \frac{\partial z_{i,2}^{(1)}}{\partial W_{3,2}^{(1)}}& \frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot \frac{\partial z_{i,3}^{(1)}}{\partial W_{3,3}^{(1)}}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot \frac{\partial z_{i,4}^{(1)}}{\partial W_{3,4}^{(1)}}\end{array}\right] \\ =\frac{1}{m}\sum _{i=1}^m\left[\begin{array}{cccc}\frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot a_{i,1}^{(0)}& \frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot a_{i,1}^{(0)}& \frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot a_{i,1}^{(0)}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot a_{i,1}^{(0)}\\ \frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot a_{i,2}^{(0)}& \frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot a_{i,2}^{(0)}& \frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot a_{i,2}^{(0)}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot a_{i,2}^{(0)}\\ \frac{\partial J}{\partial z_{i,1}^{(1)}}\cdot a_{i,3}^{(0)}& \frac{\partial J}{\partial z_{i,2}^{(1)}}\cdot a_{i,3}^{(0)}& \frac{\partial J}{\partial z_{i,3}^{(1)}}\cdot a_{i,3}^{(0)}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\cdot a_{i,3}^{(0)}\end{array}\right] \\ =\frac{1}{m}\sum _{i=1}^m\left[\begin{array}{c}{a}_{i,1}^{(0)}\\ a_{i,2}^{(0)}\\ a_{i,3}^{(0)}\end{array}\right]\left[\begin{array}{cccc}\frac{\partial J}{\partial z_{i,1}^{(1)}}& \frac{\partial J}{\partial z_{i,2}^{(1)}}& \frac{\partial J}{\partial z_{i,3}^{(1)}}& \frac{\partial J}{\partial z_{i,4}^{(1)}}\end{array}\right]=\frac{1}{m}\sum _{i=1}^m(a_i^{(0)}{)}^T\cdot \frac{\partial J}{\partial z_i^{(1)}}=\frac{1}{m}(a^{(0)}{)}^T\cdot \frac{\partial J}{\partial z^{(1)}}=R^{3\times m}\cdot R^{m\times 4}=R^{3\times 4} \end{gathered}$$

小结

TestNet网络的前向操作如下：

$$z^{(1)}=a^{(0)}\cdot W^{(1)}+b^{(1)}$$

$$a^{(1)}=relu(z^{(1)})$$

$$z^{(2)}=a^{(1)}\cdot W^{(2)}+b^{(2)}$$

$$h(z^{(2)})=\left[\begin{array}{cc}p(y_1=1)& p(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)& p(y_m=2)\end{array}\right]$$

$$J(z^{(2)})=(-1)\sum _{i=1}^m\sum _{j=1}^2\cdot 1(y_{m,j}=1)\ln p(y_{m,j}=1)$$

反向传播如下：

$$\frac{\partial J}{\partial z^{(2)}}=\left[\begin{array}{cc}p(y_1=1)-1(y_1=1)& p(y_1=2)-1(y_1=2)\\ ⋮& ⋮\\ p(y_m=1)-1(y_m=1)& p(y_m=2)-1(y_m=2)\end{array}\right]$$

$$\frac{\partial J}{\partial W^{(2)}}=\frac{1}{m}(a^{(1)}{)}^T\cdot \frac{\partial J}{\partial z^{(2)}}$$

$$\frac{\partial J}{\partial b^{(2)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_i^{(2)}}$$

$$\frac{\partial J}{\partial a^{(1)}}=\frac{\partial J}{\partial z^{(2)}}\cdot (W^{(2)}{)}^T$$

$$\frac{\partial J}{\partial z^{(1)}}=\frac{\partial J}{\partial a^{(1)}}\ast 1(z^{(1)}\ge 0)$$

$$\frac{\partial J}{\partial W^{(1)}}=\frac{1}{m}(a^{(0)}{)}^T\cdot \frac{\partial J}{\partial z^{(1)}}$$

$$\frac{\partial J}{\partial b^{(1)}}=\frac{1}{m}\sum _{i=1}^m\frac{\partial J}{\partial z_i^{(1)}}$$

假设批量数据大小为$m$，数据维数为$D$，网络层数为$L$（$1,2,...,l,...,L$），输出类别为$C$

参考反向传导算法和神经网络反向传播的数学原理，设每层输入向量为残差${\delta }^{(l)}=\frac{\partial J(W,b)}{\partial z^{(l)}}$，用于表示该层对最终输出值的残差造成的影响；而最终输出值的残差${\delta }^{(L)}$就是损失函数对输出层输入向量的梯度

前向传播执行步骤

1. 层与层之间的操作就是输出向量和权值矩阵的加权求和以及对输入向量的函数激活（以relu为例）

   $$\begin{gathered} z^{(l)}=a^{(l-1)}\cdot W^{(l)}+b^{(l)} \\ {a}^{(l)}=relu(z^{(l)}) \end{gathered}$$

2. 输出层输出结果后，进行评分函数的计算，得到最终的计算结果（以softmax分类为例）

   $$h(z^{(L)})=\left[\begin{array}{ccc}p(y_1=1)& \dots & p(y_1=C)\\ ⋮& ⋮& ⋮\\ p(y_m=1)& \dots & p(y_m=C)\end{array}\right]=\left[\begin{array}{ccc}\frac{exp(z_{1,1}^{(2)})}{\sum exp(z_1^{(2)})}& \dots & \frac{exp(z_{1,C}^{(2)})}{\sum exp(z_1^{(2)})}\\ ⋮& ⋮& ⋮\\ \frac{exp(z_{m,1}^{(2)})}{\sum exp(z_m^{(2)})}& \dots & \frac{exp(z_{m,C}^{(2)})}{\sum exp(z_m^{(2)})}\end{array}\right]$$

3. 损失函数根据计算结果判断最终损失值（以交叉熵损失为例）

   $$J(z^{(L)})=(-1)\sum _{i=1}^m\sum _{j=1}^2\cdot 1(y_{m,j}=1)\ln p(y_{m,j}=1)$$

反向传播执行步骤

1. 计算损失函数对于输出层输入向量的梯度(最终层残差)

   $${\delta }^{(L)}=\frac{\partial J}{\partial z^{(L)}}=\left[\begin{array}{ccc}p(y_1=1)-1(y_1=1)& \dots & p(y_1=C)-1(y_1=C)\\ ⋮& ⋮& ⋮\\ p(y_m=1)-1(y_m=1)& \dots & p(y_m=C)-1(y_m=C)\end{array}\right]$$

2. 计算中间隐藏层的残差值（$L-1,L-2,...1$）

   $${\delta }^{(l)}=\frac{\phi J}{\phi z^{(l)}}=(\frac{\phi J}{\phi z^{(l+1)}}\cdot \frac{\phi z^{(l+1)}}{\phi a^{(l)}})\ast \frac{\phi a^{(l)}}{\phi z^{(l)}}=({\delta }^{(l+1)}\cdot (W^{(l+1)}{)}^T)\ast 1(z^{(l)}\ge 0)$$

3. 完成所有的可学习参数（权值矩阵和偏置向量）的梯度计算

   $$\begin{gathered} {\mathrm{\nabla }}_{W^{(l)}}J(W,b)=\frac{1}{m}(a^{(l-1)}{)}^T\cdot {\delta }^{(l)} \\ {\mathrm{\nabla }}_{b^{(l)}}J(W,b)=\frac{1}{m}\sum _{i=1}^m{\delta }_i^{(l)} \end{gathered}$$

4. 更新权值矩阵和偏置向量

   $$\begin{gathered} W^{(l)}=W^{(l)}-\alpha [\mathrm{\nabla }{W}^{(l)}+\lambda W^{(l)}] \\ {b}^{(l)}=b^{(l)}-\alpha \mathrm{\nabla }{b}^{(l)} \end{gathered}$$

相关阅读

• 神经网络推导-单个数据