1
$\begingroup$

Let:

  • $x$ be a vector of dimension $n \times 1$.
  • A be a matrix of dimension $n \times m$.

I want to find the derivative of $x^T A$ w.r.t. $x$.

By both the numerator and denominator conventions, the derivative of $x^T$ w.r.t. $x$ seems to be the $n \times n$ identity matrix.

By the product rule, $d(x^T \cdot A) = d(x^T) \cdot A + x^T \cdot d(A) $.

$d(A)$ is $0$, and since $d(x^T) = I_n$, the expression should reduce to just $A$.

But Wikipedia claims it's actually $A^T$.

What did I miss/do wrong?

  • 0
    are you sure that $A$ is $n \times m$? – 2017-01-02

1 Answers 1

0

Notice that $x^TAx$ is a function $\mathbb{R}^n\to \mathbb{R}$ so the claim that the derivative is $A^T$ does not make sense. (Also $A$ has to be square for this to make sense.)

Now, $$f(x)=x^TAx=\sum\limits_{i,j}a_{i,j}x_ix_j $$ and so differentiating with respect to $x_i$ gives us $$ \frac{\partial}{\partial x_i}f(x) = \sum_{j=1}^na_{i,j}x_j + \sum_{j=1}^na_{j,i}x_j= e_i^T(A+A^T)x.$$

The derivative is then $\nabla f = x^T(A^T+A).$

  • 1
    How would I derive that in a "coordinate-free" way, in the sense that I'd rather not refer to the elements of the matrices as much as possible, since most of the point of this is to avoid messy index manipulation? – 2017-01-02
  • 0
    Although, I didn't ask for the derivative of $x^T A x$, but $x^T A$. – 2017-01-02
  • 0
    @AlokSingh "I want to find the derivative of $x^TAx$ w.r.t. $x.$" – 2017-01-02
  • 0
    In regards to your question about why you get $A$ but it should be $A^T$, it is arguably a matter of convention. The function $x^TA$ outputs row vectors, but when talking about functions between finite dimensional spaces we usually talk in column vectors. This swap in row and column vectors corresponds to taking the transpose $A^T$. So if we write the function $f(x)=x^TA$ in the usual convention we get $f(x)=A^Tx.$ – 2017-01-02
  • 0
    Oops. Fixed that. – 2017-01-03