How is it possible that the matrix equation $\textbf{Ax} = \textbf{b}$ has a least squares solution that is a line through the origin? Isn't it the point closest to the vector $\textbf{x}$ in the column space of $\textbf{A}$ thus always a one-dimensional space?
Dimension of least squares solution
-
0Who says it's a line through the origin? – 2012-11-16
1 Answers
The least squares solution $\hat{x}$ to the system $Ax = b$ can be found by solving the normal equations $A^T A \hat{x} = A^T b$. It is known that $\mathrm{rank}(A^TA) = \mathrm{rank}(A)$, so by rank-nullity, the set of solutions to this system forms an affine space (a hyperplane) which does not necessarily pass through the origin, but which, if translated to the origin, forms a subspace of dimension $n - \mathrm{rank}(A)$, where $n$ is the number of columns of $A$.
Given this, there are several things you've mentioned I need to address. Firstly, the least squares solution does not need to be a subspace. Furthermore, even if it were a subspace, it does not need to be one-dimensional (a line), as the number of least squares solutions depends on the size of the null space of $A$, i.e. there can be many choices of $\hat{x}$ such that $A\hat{x}$ is the closest point to $b$ in the column space of $A$.
-
0Hmm, it looks like I misinterpreted that the least squares solution was in the $c$olumn space but indeed, it isn't at all. – 2012-11-17