I have found something about the product rules for matrix-functions in https://ccrma.stanford.edu/~dattorro/matrixcalc.pdf $ \frac{d(f(x)^Tg(x))}{dx}=\frac{df(x)}{dx}\cdot g(x)+\frac{dg(x)}{dx}\cdot f(x) $ I verify this in the example list in http://www.psi.toronto.edu/matrix/calculus.html. For example: $ \frac{d (Ax+b)^TC(Dx+e)}{dx} = A^TC(Dx+e) + D^TC^T(Ax+b) $ $ \frac{d (x^TCx)}{dx} = (C+C^T)x $
But when I met this one, I'm confused... $ \frac{d (a^TX^TXb)}{dX} = X(ab^T + ba^T) $ Following the formula in matrixcalc.pdf, I get this $\begin{align} f(x)&=Xa\\ g(x)&=Xb\\ \frac{d (a^TX^TXb)}{dX} &= \frac{df(x)}{dx}\cdot Xb+\frac{dg(x)}{dx}\cdot Xa \\ &=a^TXb+b^TXa\end{align}$ which is different with the correct result. I don't know what I'm doing wrong, please help me.. Thanks!