5
$\begingroup$

I'm taking a machine learning course and it involves a lot of matrix computation like compute the derivatives of a matrix with respect to a vector term. In my linear algebra course these material is not covered and I browsed some book in the school library but didn't find something relevant to my problem. So can you recommend me a book which covers these matrix computations?

Thanks!

  • 0
    Related: http://math.stackexchange.com/questions/135008/a-book-for-self-study-of-matrix-decompositions2012-11-02

4 Answers 4

2

Bishop's excellent machine learning textbook, Pattern Recognition and Machine Learning, has an appendix on "Properties of Matrices". One section in this appendix (p. 697) is about Matrix Derivatives, and discusses formulas like this:

\begin{equation} \frac{\partial}{\partial A} \text{Tr}(AB) = B^T \end{equation}

and

\begin{equation} \frac{\partial}{\partial A} \ln | A | = \left(A^{-1} \right)^T, \end{equation} for example.

Peter Lax has a great book called Linear Algebra and its Applications. Chapter 9 is entitled "Calculus of Vector- and Matrix-Valued Functions".

4

I recommend two books

  1. Matrix Algebra from a statistician's perspective by David Harville. (chapter 15 covers derivatives of matrices with respect to matrices.)
  2. Matrix Differential Calculus with Applications in Statistics and Econometrics by Jan Magnus and Heinz Neudecker. (a huge portion of the book is about that particular question of differentiation.)

These two books are also excellent because they come with two additional monographs with a lot of solved problems (check for instance Matrix Algebra (Econometric Exercises, Vol. 1) by Karim Abadir and Jan Magnus which has a lot of solved problems on matrix differentiation). Also they have the additional advantage that they are directly relevant to machine learning.

3

I suggest Matrix computation by Golub and van Loan.

  • 1
    This needs about +102012-11-02
2

I find the Matrix Cookbook handy. It's not a textbook, rather, to quote the introduction:

These pages are a collection of facts (identities, approximations, inequalities, relations, ...) about matrices and matters relating to them. It is collected in this form for the convenience of anyone who wants a quick desktop reference

  • 0
    This seems good because it has a whole chapter on derivatives. "This section is covering differentiation of a number of expressions with respect to a matrix X."2012-11-06