5
$\begingroup$

I need a help with one example. I have to proove that hessian matrix of a quadratic form $f(x)=x^TAx$ is $f^{\prime\prime}(x) = A + A^T$. I am not even sure how the Jaxobian looks like (I never did one for $x \in R^n$). Thanks for any advice.

6 Answers 6

7

So let's compute the first derivative, by definition we need to find $f'(x)\colon\mathbb R^n \to \mathbb R^n$ such that $ f(x+h) = f(x) + f'(x)h + o(h), \qquad h \to 0 $ We have \begin{align*} f(x+h) &= (x+h)^tA(x+h)\\ &= x^tAx + h^tAx + x^tAh + h^tAh\\ &= f(x) + x^t(A + A^t)h + h^tAh \end{align*} As $|h^tAh|\le \|A\||h|^2 = o(h)$, we have $f'(x) = x^t(A + A^t)$ for each $x \in \mathbb R^n$. Now compute $f''$, we have \begin{align*} f'(x+h) &= x^t(A + A^t) + h^t(A + A^t)\\ &= f(x) + h^t(A + A^t) \end{align*} So $f''(x) = A + A^t$.

4

Intuitively, the gradient and Hessian of $f$ satisfy \begin{equation} f(x + \Delta x) \approx f(x) + \nabla f(x)^T \Delta x + \frac12 \Delta x^T Hf(x) \Delta x \end{equation} and the Hessian is symmetric.

In this problem, \begin{align*} f(x + \Delta x) &= (x + \Delta x)^T A (x + \Delta x) \\ &= x^T A x + \Delta x^T A x + x^T A \Delta x + \Delta x^T A \Delta x \\ &= x^T A x + \Delta x^T(A + A^T)x + \frac12 \Delta x(A + A^T) \Delta x. \end{align*}

Comparing this with the approximate equality above, we see that $\nabla f(x) = (A + A^T) x$ and $Hf(x) = A + A^T$.

3

Write explicitly $f(x)=\sum_{i,j}(\text{2nd degree monomials})$ The hessian is the matrix $H=(\partial_i\partial_jf(x)).$

2

For $f(x)=x^{\top}Ax$ where $f(x)\colon\mathbb R^n \to \mathbb R^1$, the Jacobian $f'(x)\colon\mathbb R^n \to \mathbb R^n$ can be solved as

$f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h}$

$f(x+h)=(x+h)^{\top}A(x+h)=(x^{\top}A+h^{\top}A)(x+h)=x^{\top}Ax+x^{\top}Ah+h^{\top}Ax+h^{\top}Ah$

$f(x+h)=f(x)+x^{\top}Ah+x^{\top}A^{\top}h+h^{\top}Ah=f(x)+x^{\top}(A+A^{\top})h+h^{\top}Ah$

$f'(x)=\lim_{h\to0}\frac{f(x)+x^{\top}(A+A^{\top})h+h^{\top}Ah-f(x)}{h}=\lim_{h\to0}\frac{(x^{\top}(A+A^{\top})+h^{\top}A)h}{h}$

$f'(x)=\lim_{h\to0}x^{\top}(A+A^{\top})+h^{\top}A=x^{\top}(A+A^{\top})$

Thus, the Hessian $f''(x)\colon\mathbb R^n \to \mathbb R^{n\times n}$ can be found as

$f''(x)=\lim_{h\to0}\frac{f'(x+h)-f'(x)}{h}$

$f'(x+h)=(x+h)^{\top}(A+A^{\top})=x^{\top}(A+A^{\top})+h^{\top}(A+A^{\top})$

$f''(x)=\lim_{h\to0}\frac{x^{\top}(A+A^{\top})+h^{\top}(A+A^{\top})-x^{\top}(A+A^{\top})}{h}=\lim_{h\to0}A+A^{\top}$

Finally $f''(x)=A+A^{\top}$