2
$\begingroup$

Assuming there's a variable I want to measure, but I have only very noisy instrument to do so. So I want to take multiple measurements so that I have a better chance to recover the state of the variable. Hopefully, with each measurement, my instrument can report the result as a Gaussian distribution , with the mean to be the most likely state of variable and the standard deviation suggests a rough possible region of the state.

My problem now is that I don't know how to combine these multiple measurements to get a sensible answer. My guess is that it would be nice if I can get a new gaussian from these results, with the mean centered at the expectation value of the state of the variable, and a standard deviation to reflect how confident I am about the result...

I tried to teach myself about gaussians, and probabilities, but I just couldn't get my head around...please can someone help me?

  • 0
    Sum of gaussian is a gaussian. The new gaussian will have anew mean which is the sum of the mean your previous gaussian and the variance will be the sum of variance of your previous gaussian – 2017-02-23
  • 0
    thanks for pointing it out. I just had a quick a look at the wikipedia on the sum of gaussian, seems like a solution. But maybe i missed something - say, in a 1D configuration, I take two measurements, both measurement suggest a mean of 1, if the new mean is the sum of the means then it would be 1+1=2. But wouldn't it be sensible to have a new mean of 1, because both measurements has suggested so. – 2017-02-23
  • 0
    Are you adding two gaussian noises together or adding two measurements from the same noise? I answered the former but your comment seems to indicate the latter – 2017-02-23
  • 0
    What do you mean by adding two gaussian noises? Yeah, I was thinking adding the measurement. – 2017-02-23
  • 0
    Then, what Harry49 has written describes your situation. – 2017-02-23

2 Answers 2

2

Actually, you are measuring something of the form \begin{equation} X_i = v + b_i \, , \end{equation} where $v$ is the deterministic value you want to measure, and $b_i$ is the value of a Gaussian noise at the $i$th measurement. If the measurements are independent from each other, then simply take the arithmetic mean \begin{equation} \overline{X}_n = \frac{1}{n} \sum_{i=1}^n X_i \, , \end{equation} which has a normal distribution, by linear combination of Gaussian variables.

If your system is ergodic (broadly speaking, the system and the noise do not change behavior over time, i.e. $v$ and the distribution of $b_i \sim\mathcal{N}(\mu,\sigma^2)$ do not change over time), then the expected value and the variance of $\overline{X}_n$ are \begin{equation} E(\overline{X}_n) = \frac{1}{n} \sum_{i=1}^n E(X_i) = v+\mu \, ,\\ V(\overline{X}_n) = \frac{1}{n^2} \sum_{i=1}^n V(X_i) = \frac{\sigma^2}{n} \, . \end{equation} The random variable $\overline{X}_n$ has a normal distribution $\mathcal{N}(v+\mu, \sigma^2/n)$. If you have a reference measurement where the value $v$ is known, e.g. deduced from another measurement technique, then you can estimate $\mu$ and deduce by how much the noise modifies the mean of $\overline{X}_n$.

If the noise distribution changes at each measurement, $b_i\sim \mathcal{N}(\mu_i,{\sigma_i}^2)$ for each $i$, then the arithmetic mean $\overline{X}_n$ has a normal distribution $\mathcal{N}(v + \overline{\mu}_n, {\overline{\sigma^2}}_n/n)$. Alternatively, one can compute the weighted and centered mean \begin{equation} \widetilde{X}_n = \sum_{i=1}^n w_i \left(X_i - \mu_i\right) \quad\text{with the weights}\quad w_i = \frac{{\sigma_i}^{-1}}{\sum_{j=1}^n {\sigma_j}^{-1}} \, , \end{equation} which reduces to the arithmetic mean $\overline{X}_n$ when $\mu_i = 0$ and $\sigma_i = \sigma$ for all $i$. The expected value and the variance of $\widetilde{X}_n$ are \begin{equation} E(\widetilde{X}_n) = v \sum_{i=1}^n w_i = v \, ,\\ V(\widetilde{X}_n) = \sum_{i=1}^n {w_i}^2 {\sigma_i}^2 = n \left(\sum_{j=1}^n {\sigma_j}^{-1}\right)^{-2} . \end{equation}

  • 0
    what happens if noise changes - but my system does report the confidence level for each measure. For example, the first time I get a very narrow peaky gaussian, followed by a broad flat gaussian. How do I combine these cases? – 2017-02-23
  • 0
    Then you will have to use change detection (also known as change point detection). – 2017-02-23
  • 0
    sorry, i don't understand...what does change detection tell me? the variable being measured stays still overtime; each time the instrument noise changes I know by how much it has changed, so what should i be detecting? – 2017-02-23
  • 0
    Thank you, I think things start to look good now. Just one more questions, I noticed arithmetic mean is used to calculate the final expectation value - but wouldn't it be nicer if I use weighted mean? e.g. if I have two instruments to take my measures, I want to favour the one I know is more stable (with a smaller noise value). Do you know how I can derive the weight from noise variance? – 2017-02-25
  • 0
    I edited my answer accordingly – 2017-02-25
1

I think the previous answer was clear but I was confused on why $\widetilde{X_n}$ was defined so.

Let us start afresh on the problem. So there are multiple (focus on two first) measurements and we want to aggregate it to get a better estimate on the true state (with smallest variance). Let us suppose if both measurement were independent and identical with following form. Let us derive the best linear unbiased estimator for this simple case.

$X_i = \nu + \sigma Z_i$ where $Z_i$ is standard normal (mean 0 and variance 1). So $X_i$ has mean $\nu$ and variance $\sigma^2$.

By symmetry, a natural choice for a good linear estimator is equal weight of 0.5. $$\overline{X} = \frac{X_1+ X_2}{2}$$

This estimator is unbiased as $\mathbb{E}\overline{X} = \nu$. Also the variance is $\frac{\sigma^2}{2}$ This is actually the best linear estimator as it has lowest possible variance.

Now, when the measurements have difference variances, that is when things get tricky. Let us take a simple case with two measurements as follows. $$X_1 = \nu + \sigma_1^2 Z_1$$ $$X_2 = \nu + \sigma_2^2 Z_2$$ $Z_1$ and $Z_2$ are standard normal. An unbiased estimator for the parameter $\nu$ is given by $$\widetilde{X} = w_1X_1+ (1-w_1) X_2$$ The parameter $w_1$ lies in $(0,1)$. The mean as discussed before is $\nu$. The expression for variance is given by $$Var(\widetilde{X})=w_1^2 \sigma_1^2 + (1-w_1)^2 \sigma_2^2$$ Using first order conditions, the best linear estimator (or optimal value of $w_1$) is given by $$w_1 = \frac{\sigma_2^2}{\sigma_1^2+\sigma_2^2}$$ In general, when you have multiple measurements the weights are given by,

$$w_i = \sigma_i^{-2}\left(\sum_j \sigma_j^{-2}\right)^{-1}$$

To get the expression use KKT conditions on the condition that $\sum w_i =1$

Note the previous answer has some other weights. I dont know why.