数值计算之 梯度向量和梯度矩阵，雅可比矩阵，海森矩阵

前言

非线性最小二乘中的函数求导内容，主要涉及梯度向量、雅可比矩阵和海森矩阵。因此提前做一个辨析。实际上之前在矩阵求导中已经提到过这些内容。

梯度向量

对于实值向量函数 f ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f(x)\in R,x=(x_1,x_2,\dots,x_n)^T f(x)∈R,x=(x1​,x2​,…,xn​)T，其梯度向量可表示为： ∇ f ( x ) = ∂ f ( x ) ∂ x = [ ∂ f ∂ x 1 ∂ f ∂ x 2 … ∂ f ∂ x n ] \nabla f(x)=\frac {\partial f(x)}{\partial x}= \begin{bmatrix} \frac {\partial f}{\partial x_1} \\ \frac {\partial f}{\partial x_2} \\ \dots \\ \frac {\partial f}{\partial x_n} \\ \end{bmatrix} ∇f(x)=∂x∂f(x)​=⎣⎢⎢⎢⎡​∂x1​∂f​∂x2​∂f​…∂xn​∂f​​⎦⎥⎥⎥⎤​ 梯度向量的布局与分母的布局相同，也就是分母布局。

梯度矩阵

对于实值矩阵函数 f ( X ) ∈ R f(X)\in R f(X)∈R，其梯度矩阵 ∇ f ( X ) \nabla f(X) ∇f(X)可表示为： ∇ f ( X ) = ∂ f T ( X ) ∂ X = [ ∂ f ∂ x 11 ∂ f ∂ x 12 … ∂ f ∂ x 1 n ∂ f ∂ x 21 ∂ f ∂ x 22 … ∂ f ∂ x 2 n … … … … ∂ f ∂ x n 1 ∂ f ∂ x n 2 … ∂ f ∂ x n n ] \nabla f(X)=\frac {\partial f^T(X)}{\partial X}= \begin{bmatrix} \frac {\partial f}{\partial x_{11}} & \frac {\partial f}{\partial x_{12}} & \dots & \frac {\partial f}{\partial x_{1n}} \\ \frac {\partial f}{\partial x_{21}} & \frac {\partial f}{\partial x_{22}} & \dots & \frac {\partial f}{\partial x_{2n}} \\ \dots & \dots & \dots & \dots\\ \frac {\partial f}{\partial x_{n1}} & \frac {\partial f}{\partial x_{n2}} & \dots & \frac {\partial f}{\partial x_{nn}} \\ \end{bmatrix} ∇f(X)=∂X∂fT(X)​=⎣⎢⎢⎢⎡​∂x11​∂f​∂x21​∂f​…∂xn1​∂f​​∂x12​∂f​∂x22​∂f​…∂xn2​∂f​​…………​∂x1n​∂f​∂x2n​∂f​…∂xnn​∂f​​⎦⎥⎥⎥⎤​ 对于实值向量函数 f i ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f_i(x)\in R,x=(x_1,x_2,\dots,x_n)^T fi​(x)∈R,x=(x1​,x2​,…,xn​)T组成的向量 ( f 1 ( x ) , f 2 ( x ) , … , f n ( x ) ) T (f_1(x),f_2(x),\dots,f_n(x))^T (f1​(x),f2​(x),…,fn​(x))T，其梯度矩阵可表示为： ∇ f ( x ) = ∂ f T ( x ) ∂ x = [ ∂ f 1 ∂ x 1 ∂ f 2 ∂ x 1 … ∂ f n ∂ x 1 ∂ f 1 ∂ x 2 ∂ f 2 ∂ x 2 … ∂ f n ∂ x 2 … … … … ∂ f 1 ∂ x n ∂ f 2 ∂ x n … ∂ f n ∂ x n ] \nabla f(x)=\frac {\partial f^T(x)}{\partial x} =\begin{bmatrix} \frac {\partial f_1}{\partial x_{1}} & \frac {\partial f_2}{\partial x_{1}} & \dots & \frac {\partial f_n}{\partial x_{1}} \\ \frac {\partial f_1}{\partial x_{2}} & \frac {\partial f_2}{\partial x_{2}} & \dots & \frac {\partial f_n}{\partial x_{2}} \\ \dots & \dots & \dots & \dots\\ \frac {\partial f_1}{\partial x_{n}} & \frac {\partial f_2}{\partial x_{n}} & \dots & \frac {\partial f_n}{\partial x_{n}} \\ \end{bmatrix} ∇f(x)=∂x∂fT(x)​=⎣⎢⎢⎢⎡​∂x1​∂f1​​∂x2​∂f1​​…∂xn​∂f1​​​∂x1​∂f2​​∂x2​∂f2​​…∂xn​∂f2​​​…………​∂x1​∂fn​​∂x2​∂fn​​…∂xn​∂fn​​​⎦⎥⎥⎥⎤​ 梯度矩阵需要保持分母的布局不变，也就是分母布局。

雅可比矩阵

对于实值矩阵函数 f ( X ) ∈ R f(X)\in R f(X)∈R，其雅可比矩阵 J ( X ) J(X) J(X)可表示为： J ( X ) = ∂ f ( X ) ∂ X T = [ ∂ f ∂ x 11 ∂ f ∂ x 21 … ∂ f ∂ x n 1 ∂ f ∂ x 12 ∂ f ∂ x 22 … ∂ f ∂ x n 2 … … … … ∂ f ∂ x 1 n ∂ f ∂ x 2 n … ∂ f ∂ x n n ] J(X)=\frac {\partial f(X)}{\partial X^T}= \begin{bmatrix} \frac {\partial f}{\partial x_{11}} & \frac {\partial f}{\partial x_{21}} & \dots & \frac {\partial f}{\partial x_{n1}} \\ \frac {\partial f}{\partial x_{12}} & \frac {\partial f}{\partial x_{22}} & \dots & \frac {\partial f}{\partial x_{n2}} \\ \dots & \dots & \dots & \dots\\ \frac {\partial f}{\partial x_{1n}} & \frac {\partial f}{\partial x_{2n}} & \dots & \frac {\partial f}{\partial x_{nn}} \\ \end{bmatrix} J(X)=∂XT∂f(X)​=⎣⎢⎢⎢⎡​∂x11​∂f​∂x12​∂f​…∂x1n​∂f​​∂x21​∂f​∂x22​∂f​…∂x2n​∂f​​…………​∂xn1​∂f​∂xn2​∂f​…∂xnn​∂f​​⎦⎥⎥⎥⎤​

对于由实值向量函数 f i ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f_i(x)\in R,x=(x_1,x_2,\dots,x_n)^T fi​(x)∈R,x=(x1​,x2​,…,xn​)T组成的向量 ( f 1 ( x ) , f 2 ( x ) , … , f n ( x ) ) T (f_1(x),f_2(x),\dots,f_n(x))^T (f1​(x),f2​(x),…,fn​(x))T，其雅可比矩阵可表示为： J ( x ) = ∂ f ( x ) ∂ x T = [ ∂ f 1 ∂ x 1 ∂ f 1 ∂ x 2 … ∂ f 1 ∂ x n ∂ f 2 ∂ x 1 ∂ f 2 ∂ x 2 … ∂ f 2 ∂ x n … … … … ∂ f n ∂ x 1 ∂ f n ∂ x 2 … ∂ f n ∂ x n ] J(x)=\frac {\partial f(x)}{\partial x^T}=\begin{bmatrix} \frac {\partial f_1}{\partial x_1} & \frac {\partial f_1}{\partial x_2} & \dots & \frac {\partial f_1}{\partial x_n} \\ \frac {\partial f_2}{\partial x_1} & \frac {\partial f_2}{\partial x_2} & \dots & \frac {\partial f_2}{\partial x_n} \\ \dots & \dots & \dots & \dots \\ \frac {\partial f_n}{\partial x_1} & \frac {\partial f_n}{\partial x_2} & \dots & \frac {\partial f_n}{\partial x_n} \\ \end{bmatrix} J(x)=∂xT∂f(x)​=⎣⎢⎢⎢⎡​∂x1​∂f1​​∂x1​∂f2​​…∂x1​∂fn​​​∂x2​∂f1​​∂x2​∂f2​​…∂x2​∂fn​​​…………​∂xn​∂f1​​∂xn​∂f2​​…∂xn​∂fn​​​⎦⎥⎥⎥⎤​ 雅可比矩阵需要保持分子的布局不变，也就是分子布局。

海森矩阵

对于实值向量函数 f ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f(x)\in R,x=(x_1,x_2,\dots,x_n)^T f(x)∈R,x=(x1​,x2​,…,xn​)T，其海森矩阵 H ( x ) H(x) H(x)是函数的二阶导，实际上就是 f ( x ) f(x) f(x)的梯度向量对 x x x的雅可比矩阵：

H ( x ) = J ( ∇ f ( x ) ) = ∂ ∂ f ( x ) ∂ x ∂ x T = [ ∂ 2 f ∂ x 1 ∂ x 1 ∂ 2 f ∂ x 1 ∂ x 2 … ∂ 2 f ∂ x 1 ∂ x n ∂ 2 f ∂ x 2 ∂ x 1 ∂ 2 f ∂ x 2 ∂ x 2 … ∂ 2 f ∂ x 2 ∂ x n … … … … ∂ 2 f ∂ x n ∂ x 1 ∂ 2 f ∂ x n ∂ x 2 … ∂ 2 f ∂ x n ∂ x n ] H(x)=J(\nabla f(x))=\frac{\partial \frac {\partial f(x)}{\partial x}}{\partial x^T} = \begin{bmatrix} \frac {\partial^2f}{\partial x_1\partial x_1} & \frac {\partial^2f}{\partial x_1\partial x_2} & \dots& \frac {\partial^2f}{\partial x_1\partial x_n} \\ \frac {\partial^2f}{\partial x_2\partial x_1} & \frac {\partial^2f}{\partial x_2\partial x_2} & \dots& \frac {\partial^2f}{\partial x_2\partial x_n} \\ \dots & \dots & \dots & \dots \\ \frac {\partial^2f}{\partial x_n\partial x_1} & \frac {\partial^2f}{\partial x_n\partial x_2} & \dots& \frac {\partial^2f}{\partial x_n\partial x_n} \\ \end{bmatrix} H(x)=J(∇f(x))=∂xT∂∂x∂f(x)​​=⎣⎢⎢⎢⎡​∂x1​∂x1​∂2f​∂x2​∂x1​∂2f​…∂xn​∂x1​∂2f​​∂x1​∂x2​∂2f​∂x2​∂x2​∂2f​…∂xn​∂x2​∂2f​​…………​∂x1​∂xn​∂2f​∂x2​∂xn​∂2f​…∂xn​∂xn​∂2f​​⎦⎥⎥⎥⎤​

总结

梯度向量和梯度矩阵保持分母布局不变；

雅可比矩阵保持分子布局不变；

海森矩阵其实是实值向量函数的梯度向量对自变量求雅可比矩阵。

补充

海森矩阵是对称矩阵。因为当一阶偏导连续，并且二阶混合偏导连续，则二阶混合偏导与求导顺序无关。