0
$\begingroup$

I have a vector of values $\vec d$ that have each been measured with known uncertainty $\vec \sigma$.

With an equation

$$ \mathbf{A} \vec x = \vec d$$

where both $\mathbf{A}$ and $\vec d$ have been divided by $\vec \sigma$, I can estimate $\vec x$ using the singular value decomposition (SVD) via

$$ \mathbf{A} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^T $$ $$ \vec x = \mathbf{V} \boldsymbol{\Sigma}^{-1} \mathbf{U}^T \vec{d}. $$

What is the uncertainty on each component of $\vec x$?

  • 0
    Is $A$ invertible (i.e. square with full rank)? If not, is it deficient in row rank or in column rank or in both? (I ask mainly for the purpose of ensuring the problem makes sense. The problem of understanding the uncertainty is simple provided that the noises are uncorrelated. Then the variance of the "signal" noise is obtained in the usual way: $Var(\sum a_i X_i)=\sum_i a_i^2 Var(X_i)$.) If the noises might be correlated then the situation is more complicated but still tractable. – 2017-02-07
  • 0
    Hi, there are more rows than columns in $\mathbf{A}$. – 2017-02-07

1 Answers 1

1

I'm adding some assumptions to the question:

Assume $A$ is a $n \times n$ invertible matrix and $d$ is a random vector with covariance matrix $C$ whose diagonal is the vector $\sigma^2$. Then we seek the variance of $x$, the unique solution to $Ax=d$. But $x=A^{-1}d=A^{-1}(\mu+C^{1/2}\epsilon)$ where $\epsilon$ is the noise vector, assumed uncorrelated and with unit variance (but no other assumptions are required). Now $A^{-1} \mu$ is a fixed vector, so the covariance is that of the random vector $A^{-1} C^{1/2} \epsilon$.

If we now assume additionally that $C$ is diagonal (i.e. the noise to $d$ is uncorrelated) then we can read off the variance from here: the variance of $x_i$ is $\sum_{j=1}^n (A^{-1})_{ij}^2 \sigma^2_j$, so the standard deviation of $x_i$ is the square root of that. I think you can proceed similarly when $C$ is not diagonal but the situation will get significantly more complicated.

I'm not sure getting the SVD of $A$ involved here does you any good from the mathematical perspective (as opposed to the numerical perspective).

Apparently the $A$ here is actually $m \times n$ with $m>n$, so that there is no solution to $Ax=d$ for most $d$. Then the least squares solution is given by $x=A'd$ where $A'$ is the Moore-Penrose pseudoinverse. One can repeat the above analysis exactly as written with $A'$ replacing $A^{-1}$ and the result turns out the same.

  • 0
    Hi, $\mathbf{A}$ is not a square matrix and as such is not directly invertible. Furthermore $\mathbf{A}$ is ill-conditioned so the Moore-Penrose pseudo-inverse isn't a good option. Using the SVD skirts these issues and allows me to get good estimates of $\vec x$. – 2017-02-07
  • 0
    @rhombidodecahedron Alright, well, mathematically speaking the same principle applies: if $x$ is your least squares solution, given by $A'd$, then you have $x=A'(\mu+C^{1/2} \epsilon)$, so the variance of $x_i$ is $\sum_{j=1}^n (A')_{ij}^2 \sigma_j^2$. The $A'$ is, again mathematically speaking, the Moore-Penrose pseudoinverse of $A$; representing this formula directly in terms of the SVD will be quite cumbersome I think. How you deal with this in numerics is up to you. – 2017-02-07
  • 0
    thanks for your answer. One last thing I am wondering: why do the residuals ($A^{-1}x-d$) not play a role here? My understanding of e.g. OLS is that the uncertainty of $x$ depend on the goodness of fit. – 2017-02-14
  • 0
    @rhombidodecahedron Because I'm describing the uncertainty in what the least square solution is. You might consider a different model in which the uncertainty depends on the goodness of fit; I don't know what's popular out in industry unfortunately. – 2017-02-14
  • 0
    sorry by OLS I meant "ordinary least squares", like if you have y=mx+b, the uncertainty on m and b have some relation to the residuals y-(mx+b) don't they? – 2017-02-14