4
$\begingroup$

I have something like this $X^T A^T A X - X^T A^T B - B^T A X$ or $X^T A^T A X -2 X^T A^T B$ since X and B are really vectors ($X^TA^TB=B^TAX$).

I’d like to find the X that minimizes the expression. To do that I want to have something like this $[X-(A^TA)^{-1}A^TB]^T(A^TA)[X-(A^TA)^{-1}A^TB]-B^TA(A^TA)^{-1}A^TB$ so that the X that minimizes the expression is $(A^TA)^{-1}A^TB$.

My question is: how does all this relate to the technique of completing the squares? How can I show the squares?

To further explain my question, it’s easy to say, if you have $a^2 + 2ab$ just add and subtract $b^2$. But how can I abstract that to matrices?

PS: I assume $A^TA$ is non singular, positive definite.

  • 0
    In my question there is not a problem to be solved, more like a solution to be explained. Going to the first equation to the second equation (the long one) causes me much trouble.2011-06-12

1 Answers 1

5

Write $X^T A^T A X - X^T A^T B - B^T A X = (A X - B)^T (A X - B) - B^T B.$

So you want to minimize $(A X - B)^T (A X - B)$.

Note that $A (A^T A)^{-1} A^T$ is the orthogonal projection on ${\rm Ran}(A)$. Thus $B = A u + v$ where $u = (A^T A)^{-1} A^T B$, and $v \perp A x$ for all $x$.

Now $A X - B = A(X - u) - v$, so $(A X - B)^T (A X - B) \ge v^T v$. Thus the minimum value of $(A X - B)^T (A X - B)$ is $v^T v$, attained for $X = u$.