4
$\begingroup$

Refering to this wikipedia page Unbiased estimation of standard deviation, it says that "it follows from Jensen's inequality that the square root of the sample variance is an underestimate".

I do know that for the concave square root function, Jensen's inequality says that the square root of the mean > mean of the square root.

So, how do we conclude that the square root of the sample variance underestimates population standard deviation?

Since we know from Jensen's inequality that square root of the mean > mean of the square root, does "square root of sample variance" somehow relate to "mean of the square root" while "population standard deviation" somehow relates to "square root of the mean"?

Added after joriki's response:

Given joriki's response about using a single sampling of data, I am now left with why $s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N{(x_i-\overline{x})^2}}$ will underestimate pop std dev. In order to use Jensen's inequality (mean of the square root < square root of the mean). I need to somehow relate the expression for $s$ to "mean of square root". I do see the square root sign in the expression for $s$ but where is the "mean" of this square root quantity?

  • 3
    Yes, you have it right. Recall that the sample *variance* is an *unbiased* estimator of the variance. This means that the expectation (mean) of $\frac{1}{n-1}\sum(X_i-\bar{X})^2$ is $\sigma^2$. The square root of this is $\sigma$, the right thing. Thus $\sigma$ is greater than the expectation of $\sqrt{\frac{1}{n-1}\sum(X_i-\bar{X})^2}$, which is the expectation (mean) of the sample standard deviation. If we use the expression just mentioned as our estimator for $\sigma$, then on average this estimator will be too small.2012-02-08
  • 0
    So, do you mean that if I were to perform an experiment where I take many samplings from a population: if I first compute the unbiased estimator for population variance from each sampling; then get the mean of all these values; followed by taking the square root of this mean; I will get the unbiased estimator for population standard deviation?2012-02-08
  • 0
    JohnC: It would not be an *estimator*. An unbiased estimator is a random variable whose mean is the right thing. You have a random variable the *square root* of whose mean is the right thing.2012-02-08
  • 0
    @JohnC: That two-stage process (dividing your total sample into several samples, computing the sample variance for each and then averaging) doesn't improve the estimation; in fact it makes it worse, since you're throwing away information about how much the groups deviated with respect to each other. You'll get the best estimate if you treat all your data as one big sample. The problem of the square root skewing the result is independent of this; approaches to dealing with it are discussed in the Wikipedia article you linked to.2012-02-08
  • 0
    @Andre: I now understand that s^2 is a RV whose mean is the population variance while the value I obtain in my experiment would be a realized value of this RV. Thanks for making this distinction.2012-02-08
  • 0
    @joriki: that first part about taking all data as one big single sampling makes a lot of sense! Thanks! I know how to deal with the skewing (since it is in the article I linked). What I do not know is WHY the skewing exists? Please see my edited question.2012-02-08
  • 1
    @JohnC: You ask where is the mean. Suppose we use the already square rooted expression as an *estimator* of the population standard deviation $\sigma$. We imagine taking the mean in order to find out whether this estimator is biased or not. If the mean of this estimator is $\sigma$, then it is unbiased. The Jensen's Inequality argument shows that except in trivial cases, the mean is *less* than $\sigma$. This lets us conclude that the estimator is not unbiased, and even more, in what direction it errs on average.2012-02-08

3 Answers 3