Transpose Rules \left(AB\right)^{T} = B^{T}A^{T} \left(a^{T}Bc\right)^{T} = c^{T} B^{T}a a^{T}b = b^{T}a \left(A+B\right)C = AC + BC \left(a+b\right)^{T}C = a^{T}C + b^{T}C AB \neq BA Derivative Scalar derivative Vector derivative f\left(x\right) \to \pdv{f}{x} f\left(x\right) \to \pdv{f}{x} bx \to b x^{T}B \to B bx \to b x^{T}b \to b x^{2} \to 2x x^{T}x \to 2x bx^{2} \to 2bx x^{T}Bx \to 2Bx Products \begin{equation} \pdv{AB}{A} = B^{T}, \pdv{AB}{B} = A^{T} \end{equation}
\begin{equation} \pdv{Ax}{A} = x^{T}, \pdv{Ax}{x}= A \end{equation}
Vector and Quadratic Forms \begin{equation} \pdv{y^{T} x}{x} = y, \pdv{y^{T} x}{y} = x \end{equation}
\begin{equation} \pdv{x^{T}Ax}{x} = \left(A+A^{T}\right)x = 2Ax \end{equation}
for symmetric A Chain Rule for Matrix Multiplication Suppose:
\begin{equation} z = Wu + b \end{equation}
\begin{equation} J = J\left(z\right) \end{equation}
then: \pdv{J}{W} = \pdv{J}{z} u^{T}. Rest of them