2
$\begingroup$

So I understand the proofs behind Singular Value Decomposition but I'm having trouble interpreting it in the context of a real world problem.

Specifically, If I'm given an $m\times n$ data matrix A, where we have m training examples and n features collected for each example, I'm having trouble understanding the meaning behind Av$_j$ = $\sigma$$_j$u$_j$ where L$_A$ (left multiplication by A) is our linear transformation and $\beta$ = {v$_1$, v$_2$, ... , v$_n$} is an orthonormal basis for F$^n$ and $\gamma$ = {u$_1$, u$_2$, ... , u$_m$} is an orthonormal basis for F$^m$.

From reading various posts and articles, the idea seems to be a larger $\sigma$$_j$ indicates more variation in the data along that vector u$_j$ while a smaller variation in a certain direction u$_j$ is captured with a smaller $\sigma$$_j$.

However, when we are looking at Av$_j$ = $\sigma$$_j$u$_j$ i'm not sure why we care about what L$_A$ is doing. After all, this relationship would be great if I wanted to see what L$_A$ does when it acts on a orthonormal basis $\beta$ but A is just a data matrix so i'm not sure how to interpret the range of a data matrix or what types of transformations A is making when presented a vector x to 'do' left multiplication on.

  • 0
    If you use a matrix just to store some data and not multiply it to anything, then SVD would be a useless concept – 2017-01-06

1 Answers 1

2

Your data sets are the rows of $A$. Thus you get the $k$th sample by computing $e_k^TA$. Using the SVD this is also $$ e_k^TA=\sum_{j=1}^nσ_j(e_k^T{\bf u}_j)\,{\bf v}_j^T $$ You can reduce this sum to the leading $d$ terms with the largest $d$ singular values to get a good approximation of the data, which means that your data vectors are all close to the subspace spanned by ${\bf v}_1,…,{\bf v}_d$ for some suitably chosen $d

  • 0
    Thank you for the response. Would you be able to explain or derive that equation a little further? I'm not quite seeing the equality – 2017-01-06
  • 1
    The SVD is $A=U\Sigma V^T=\sum_jσ_j u_jv_j^T$. Now left-multiply with a canonical basis vector to extract the corresponding row... – 2017-01-06
  • 0
    thank you! haven't seen that matrix multiplication representation often. So at each iteration through j = 1 to n we are adding another term of the dot product for each space of the final mxn matrix, A. though here we restrict it to an individual row in the final matrix. so from what i've read if someone were to employ SVD they would set the smallest singular values to 0 then recompute the matrix, resulting in hopefully a lower rank matrix. But the resulting row here will still be the same size just with smaller values since some of the later sums in the dot product have been scaled by 0. – 2017-01-06
  • 0
    how would the smaller values result in a lower rank? – 2017-01-06
  • 0
    ah never mind, since we changed some of the smaller sigmas to 0, each row is now produced by fewer linearly independent vectors, v_i. In the case of the new data matrix each row is now a sum of d linearly independent vectors where d < n. hence we have reduced the rank of our matrix since rank = row rank. @LutzL would be great if you could confirm or reject my understanding. thanks! – 2017-01-06
  • 1
    Yes, that is correct. You can also define the rank as the lowest number of dyadic products in a sum that equals the matrix. Thus the rank of the reduced SVD is $d$. - It is not guaranteed that the entries of the reduced matrix are smaller than the entries of the original matrix. – 2017-01-08
  • 0
    or alternatively for looking at new rank achieved: we have for y $\in$ range of L$_A$ we have L$_A$(x) = y where x = a$_1$v$_1$ + ... a$_n$v$_n$ implies that L$_A$(x) = a$_1$*L$_A$(v$_1$) + ... a$_n$*L$_A$(v$_n$) which equals a1* $\sigma$$_1$u$_1$ + ... an*$\sigma$$_n$u$_n$. so if we made d of the latter sigmas = 0, we would reduce the dimension by n-d since we limit the 'directions' in which the range can span. sorry for the barrage of comments. this was my last, would also be great if you could confirm or reject my reasoning. @LutzL – 2017-01-08