I have been struggling with a practice problem for some time now. Here is the problem:
$f(x) = |Ax-b|^{2}$
$A$ is an $m\times n$ matrix with zero null space, b is an $m$ dimensional vector and $x$ is an n dimensional vector.
Find the hessian, and prove it is positive definite. Then, find a closed form expression for the global minimizer $x^{*}$. Some other questions but these are the ones causing problems for me.
I am struggling a lot with this question. My approach was as follows:
Write $f(x)$ without the matrices and vectors. So
$f(x) = (\sum_{k=1}^{n}a_{1k}x_{k}-b_{1})^{2} +...+(\sum_{k=1}^{n}a_{mk}x_{k}-b_{m})^{2}$
Then I computed the hessian by just taking second derivatives. It looks like this (confirm/deny?)
$\begin{bmatrix} \sum_{l=1}^{m}a_{l1}^{2} . .................... \sum_{l=1}^{m}a_{l1}a_{ln} \\ \sum_{l=1}^{m}a_{l1}a_{l2} ............................. \\ ....................................... \\ ........................................\\ ..........................\sum_{l=1}^{m}a_{ln}^{2} \end{bmatrix}$
Assuming this is correct, I am unsure of how to show it is positive definite. (Previously I've mostly used Sylvester's Criterion to show this but it doesnt seem to work here - not easily anyways)
I am also unsure of how to find a closed form for the minimizer.
Any help would be appreciated. Thanks a lot