0
$\begingroup$

I've been asked to define an estimator, $\Theta_c$ for $\sigma^2$, where $c>0$ is a constant.

$\Theta_c = c \sum_{i=1}^n(X_i-\bar{X})^2$

The question is to calculate Var($\Theta_c$) in terms of the central moments $\mu_k'=E[(X_1-\mu)^k],k>0$ and find the value of $c$ that minimizes Var($\Theta_c$). I'm having difficulty expressing the variance in terms of the central moments. I think I'm on the right path, I've rewritten in terms of $Z_i = X_i - \mu$ to eliminate the off center mean. Then $\Theta_c = c\sum_{i=1}^n(X_i-\bar{Z})^2$ and the $k$th moment of $Z_i$ will be the $k$th central moment of $X_i$, but I can't figure out how to use this to express the variance in terms of the central moments.

  • 0
    Not understanding minimizing the variance over $c.$ As $c\to 0$ the variance of the estimator (and the estimator itself) goes to $0.$ Maybe they mean minimize the mean-squared error?2017-02-08
  • 0
    Yes, I'm trying to minimize the MSE in the next part, any idea how to do this?2017-02-08

1 Answers 1

1

The trick is to write $$ \Theta_c = c\sum_k((X_k-\mu)+(\mu-\bar X))^2\\=c\sum_k((X_k-\bar X)^2-2(X_k-\mu)(\bar X-\mu)+(\bar X-\mu)^2)\\=c\left(\left(\sum_k (X_k-\mu)^2\right)-2n(\bar X-\mu)(\bar X-\mu)+n(\bar X-\mu)^2\right)\\ =c\left(\left(\sum_k (X_k-\mu)^2\right)-n(\bar X-\mu)^2\right).$$

We would now like to take the expectation. Since the variables are i.i.d. (I assume) the expectation of the first term in parentheses is easy: it's just $n\sigma^2.$ The second term is easy when you remember that the variance of the sum is the sum of the variance. I'll do it out the long way though, to prepare for later in the calculation. We have $$(\bar X-\mu)^2 = \frac{1}{n^2}\left(\sum_{k}(X_k-\mu)\right)^2 = \frac{1}{n^2}\sum_{k,l} (X_k-\mu)(X_l-\mu) \\= \frac{1}{n^2}\left(\sum_k(X_k-\mu)^2 + \sum_{k\ne l}(X_k-\mu)(X_l-\mu)\right).$$

When we take the expected value, all the terms in the second sum are zero cause they're the product of independent, zero mean variables, so we have $$ E((\bar X-\mu)^2) = \frac{1}{n}\sigma^2.$$

Putting it all together, $$ E(\Theta_c) = c(n-1)\sigma^2.$$

To get the variance, we need to square $\Theta_c,$ giving $$ \Theta_c^2 = c^2\left(\left(\sum_k (X_k-\mu)^2\right)^2 - 2n(\bar X-\mu)^2\sum_k (X_k-\mu)^2 + n^2(\bar X-\mu)^4 \right)$$

We can consider each of the terms. For the first, $$ \left(\sum_k (X_k-\mu)^2\right)^2 = \sum_{k,l}(X_k-\mu)^2(X_l-\mu)^2 = \sum_k(X_k-\mu)^4 + \sum_{k\ne l}(X_k-\mu)^2(X_l-\mu)^2.$$ Taking the expected value and seeing that there are $n$ terms in the first sum and $n(n-1)$ in the second gives $$E\left[\left(\sum_k (X_k-\mu)^2\right)^2\right] = n\mu'_4 + n(n-1)\sigma^4.$$

The next term is $$ (\bar X-\mu)^2\sum_k (X_k-\mu)^2 = \frac{1}{n^2}\sum_{k,l,m}(X_k-\mu)(X_l-\mu)(X_m-\mu)^2 \\ = \frac{1}{n^2}\left(\sum_{k,m}(X_k-\mu)^2(X_m-\mu)^2 + \sum_{(k\ne l), m} (X_k-\mu)(X_l-\mu)(X_m-\mu)^2\right).$$

Again, when we take the expected value, the second term vanishes. The first term we recognize from before. So we get $$ E\left[ (\bar X-\mu)^2\sum_k (X_k-\mu)^2 \right]= \frac{1}{n}\mu'_4 + \frac{n-1}{n}\sigma^4$$

Finally, the third term is $$ (\bar X-\mu)^4 = \frac{1}{n^4} \sum_{k,l,m,n}(X_k-\mu)(X_l-\mu)(X_m-\mu)(X_n-\mu) \\ =\frac{1}{n^4}\left( \sum_k(X_k-\mu)^4+ 3\sum_{k\ne l} (X_k-\mu)^2(X_l-\mu)^2 + \mbox{ other terms}\right)$$ where the 'other terms' are going to have zero expectation. The factor of $3$ is from the three different ways of choosing which terms to pair. We take the expectation to get $$ E\left[(\bar X-\mu)^4\right] = \frac{1}{n^3}\mu'_4+3\frac{n-1}{n^3}\sigma^4.$$

Putting the three terms together gives $$ E(\Theta_c^2) = c^2\left(\left(n-2+\frac{1}{n}\right)\mu'_4 + \left(n-2+\frac{3}{n}\right)(n-1)\sigma^4\right)$$ and we can find the variance is $$ \mathrm{Var}(\Theta_c) = E(\Theta_c^2)-E(\Theta_c)^2 =c^2\left(\left(n-2+\frac{1}{n}\right)\mu'_4 - \left(1-\frac{3}{n}\right)(n-1)\sigma^4\right).$$

Of course, as I mentioned in a comment, to minimize the variance of the estimator, take $c\to 0.$ Of course even though it's low variance, this will not be a good estimator since it's high bias.

A better thing to minimize is the mean-squared error, $$ E((\Theta_c-\sigma^2)^2).$$ To compute the mean-squared error, it's customary to write $$ E[(\Theta_c-\sigma^2)^2] = E[((\Theta_c-E(\Theta_c))-(\sigma^2-E(\Theta_c)))^2] \\ = E[(\Theta_c-E(\Theta_c))^2] +E[\Theta_c-E(\Theta_c)](E(\Theta_c)-\sigma^2) + (E(\Theta_c)-\sigma^2)^2 \\= \mathrm{Var}(\Theta_c) + (E(\Theta_c)-\sigma^2)^2 $$ so that it is seen that it is composed of a variance term and a bias-sqaured term.

Plugging in what we've computed, $$ E[(\Theta_c-\sigma^2)^2]=c^2\left(\left(n-2+\frac{1}{n}\right)\mu'_4 - \left(1-\frac{3}{n}\right)(n-1)\sigma^4\right) + ((c(n-1)-1)\sigma^2 )^2 \\ = c^2\left(\left(n-2+\frac{1}{n}\right)\mu'_4 + \left(n-2+\frac{3}{n}\right)(n-1)\sigma^4\right) -2c(n-1)\sigma^4+\sigma^4.$$

Minimizing over c gives $$ c_m = \frac{(n-1)\sigma^4}{\left(n-2+\frac{1}{n}\right)\mu'_4 + \left(n-2+\frac{3}{n}\right)(n-1)\sigma^4}.$$

For a Gaussian (i.e. with $\mu'_4 = 3\sigma^4$) this gives a minimum mean-squared error variance estimator of $$\frac{1}{n+1} \sum_k(X_k-\bar X)^2.$$

(This isn't necessarily the minimum MSE estimator overall, just the minimum of estimators that are proportional to the sample variance. Notice that it's smaller than the unbiased estimator which has coefficient $c=\frac{1}{n-1}$.)