Standard Least-Squares Minimisation

Question

Minimise the residual error gives us the standard least-squares problem: \begin{equation} \text{arg min}_x||Ax-b||^2 \end{equation}

First we rewrite the above as follows: \begin{equation} F(x)=(Ax-b)^T(Ax-b)\\ F(x)= (x^TA^T-b^T)(Ax-b)\\ F(x)= x^TA^TA x- x^TA^Tb-b^TAx+b^Tb\\ F(x) = x^TA^TA x- 2x^TA^Tb+b^Tb \end{equation}

Question 1: Why $x^TA^Tb=b^TAx$ ?

To derive the solution $x$, (with help) I calculated the gradient of the above: \begin{equation} \nabla F(x)= 2x^TA^TA-2A^Tb \end{equation}

\begin{equation} \nabla F(x)=0 \Rightarrow x^TA^TA=A^Tb \end{equation} Question 2: How can I prove that $x=(A^TA)^{-1}A^Tb$

Answer 1

For your first question, realize $b^TAx=(x^TA^Tb)^T$, and $b^TAx$ is just a number, i.e. $1\times 1$ matrix. Thus, it is equal to its transpose.

For your second question, just transpose everything again, then you'll have $$A^TAx = (x^TA^TA)^T=(A^Tb)^T=b^TA\implies x=(A^TA)^{-1}b^TA$$