0
$\begingroup$

Suppose I start with a data matrix $A=\begin{pmatrix} 1 &2 &1 \\ -2&-3 &1 \\ 3& 5&0 \end{pmatrix}$ and I replace every row by its more compact representation in the basis $ \{(1,2,1), (2,3,-1)\}$ of the row space.

That way I obtain a new matrix $B=\begin{pmatrix} 1 &0 \\ 0& -1\\ 1& 1 \end{pmatrix}$.

    1. What is the relation between the matrix $B$ and the original matrix $A$, in terms of matrix multiplication?
    1. Conceptually, I want to say that what was done is to replace the original columns (features) by new features derived from the previous ones. But is that really what was done? How do I write the new columns as a linear combination of the old columns? (In general)

2 Answers 2

2

$A=BR$, where $R$ is the matrix with the row space basis vectors of $A$ as rows. If you want to find coefficients that let you write the columns of $B$ as linear combinations of columns of $A$, you can simply solve the system $A\mathbf x=B_k$ for each column $B_k$, which you can do en masse by row-reducing the augmented matrix $[\begin{array}{c|c}A&B\end{array}]$, but since we know that $A$ doesn’t have full rank, you’ll get either an infinite number of solutions or none.

1

We have the matrix equation in $\mathrm X \in \mathbb R^{3 \times 2}$

$$\mathrm A \mathrm X = \mathrm B$$

where $\mathrm A \in \mathbb R^{3 \times 3}$ and $\mathrm B \in \mathbb R^{3 \times 2}$ are given. Since $\mathrm A$ does not have full rank, there should be either infinitely many solutions or none. The least-squares solution is

$$\hat{\mathrm X} := \arg \min \| \mathrm A \mathrm X - \mathrm B \|_{\rm F}^2 = (\mathrm A^{\top} \mathrm A)^+ \mathrm A^{\top} \mathrm B = \lim_{t \to 0} \, (\mathrm A^{\top} \mathrm A + t \, \mathrm I_3)^{-1} \mathrm A^{\top} \mathrm B = \color{blue}{\begin{bmatrix}0 & \frac{1}{7}\\ \frac{1}{5} & \frac{4}{35}\\ \frac{3}{5} & - \frac{13}{35}\end{bmatrix}}$$

>>> from sympy import *
>>> A = Matrix([[ 1, 2, 1],
                [-2,-3, 1],
                [ 3, 5, 0]])
>>> B = Matrix([[ 1, 0],
                [ 0,-1],
                [ 1, 1]])
>>> t = Symbol('t')

Let's compute the least-squares solution:

>>> X_LS = simplify((A.T * A + t*eye(3))**-1 * A.T * B)
>>> X_LS
Matrix([
[      4*t/(t**2 + 54*t + 105),   5*(t + 3)/(t**2 + 54*t + 105)],
[7*(t + 3)/(t**2 + 54*t + 105), 4*(2*t + 3)/(t**2 + 54*t + 105)],
[ (t + 63)/(t**2 + 54*t + 105),   -(t + 39)/(t**2 + 54*t + 105)]])
>>> X_LS.subs(t,0)
Matrix([
[  0,    1/7],
[1/5,   4/35],
[3/5, -13/35]])

Let's verify if the least-squares solution is indeed a solution to the original matrix equation:

>>> A * X_LS.subs(t,0) - B
Matrix([
[0, 0],
[0, 0],
[0, 0]])