Matrix and Vector Calculus

Diagonal

Prop For any matrix , pre-multiply by a diagonal matrix scales its the rows, while post-multiply by a diagonal matrix scales its the columns.

Inverse

Def Pseudo-Inverse The Pseudo-Inverse of some matrix is defined as . Specifically, if is invertible, we have

Trace

Def Trace The trace of a square matrix is the sum of diagonal elements:

Prop Trace satisfies the following properties:

• .
• .
• if is square then .

Factorizations

Thrm LU Factorization Every non-singular matrix can be factored as where is a permutation matrix, is unit lower triangular, and upper triangular and non-singular.

Thrm Cholesky Factorization Every symmetric positive definite matrix can be factored as where is lower triangular and non-singular with positive diagonal elements.

Thrm Singular Value Decomposition Any matrix can be decomposed as

Differentiation

Def Jacobian Suppose is a function. Then the gradient of with respect to is the matrix of partial derivatives called Jacobian:

Def Hessian Suppose that is a function. Then the hessian of with respect to is the matrix of second partial derivatives:

Prop Hessian is symmetric.

Thrm Differentiation Laws

• Constant:
• Addition:
• Matrix multiplication:
• Element-wise multiplication:
• Transpose:
• Inverse:
• Trace:
• Derivative:
• Determinant:
  • Specially, if is invertible, .

e.g. Suppose . Consider . Then we have

Block Matrix

Thrm Block matrix Determinant

Thrm Block Matrix Inverse