What are the nature and purpose of confidence intervals?
What is a confidence interval?
-
0Very good question, that I hope people who ask a technical confidence interval question gets directed to. – 2012-09-08
2 Answers
Consider the normal distribution $N(\mu,\sigma^2)$. Its probability density function is $x\mapsto\text{constant}\cdot\exp\left(\frac{-1}{2}\cdot\left(\frac{x-\mu}{\sigma}\right)^2\right)$. It has expected value $\mu$ and standard deviation $\sigma$. It puts probability about $0.95$ in the interval whose endpoints are $\mu\pm1.96\sigma$.
Suppose one cannot observe $\mu$ and must estimate based on a sample of $n_1$ independently chosen observations from a population with this distribution. One can show that the sample mean $\bar{X}=(X_1+\cdots+X_{n_1})/n_1$ has a normal distribution with mean $\mu$ and standard deviation $\sigma/\sqrt{n_1}$.
For the moment we assume, unrealistically, that $\sigma$ is known.
We have $ \Pr\left(\mu-1.96\frac{\sigma}{\sqrt{n_1}} < \bar X < \mu+1.96\frac{\sigma}{\sqrt{n_1}}\right) = 0.95 $ and hence $ \Pr\left( \bar X - 1.96\frac{\sigma}{\sqrt{n_1}} <\mu< \bar X + 1.96\frac{\sigma}{\sqrt{n_1}}\right) = 0.95. $ The interval whose endpoints are $ \bar X \pm 1.96\frac{\sigma}{\sqrt{n_1}}\tag{1} $ is a $95\%$ confidence interval for $\mu$.
It is tempting to say that if we observe that the two numbers $\bar X\pm 1.96\dfrac{\sigma}{\sqrt{n_1}}$ are, for example, $5$ and $9$, then $ \Pr\left(5<\mu<9\right)=0.95.\tag{DANGER!!} $ But suppose we take a second sample, this time of size $n_2$. We can form another confidence interval, using the new sample mean in the role of $\bar X$ and $n_2$ in the role of $n_1$ in $(1)$ above. What changes when we take a new sample is the endpoints of the interval. What does not change is $\mu$. The $0.95$ probability means that $95\%$ of the time, when we take another such sample, $\mu$ will be within the interval that we get. Some people take this fact to be an objection to the statement labeled "DANGER!!" above, saying that "$\Pr\left(5<\mu<9\right)=0.95$" should be considered true only if it is the case that $95\%$ of all values of $\mu$ are between $5$ and $9$, and that is clearly false, since there is only one value of $\mu$. This, however, depends on the meaning of probability. Even if one so defines probability that this objection is not valid, nonetheless the mathematics of probability do not lead to the conclusion that the statement labeled "DANGER!!" is true. In practice, though, people often act as if one should be $95\%$ sure of the statement $5<\mu<9$, given the evidence of the sample.
To say that one is "$95%$ confident" that $\mu$ lies within the confidence interval, as a term of art in statistics, means precisely that $95\%$ of the time, when one takes a new random sample of one or more observations, the interval one gets will contain $\mu$.
It was of course unrealistic to assume we know the population S.D. $\sigma$ but we have to estimate $\mu$ based on a sample. Suppose we estimate $\sigma$ based on the sample by using the square root of $ S^2 = \frac{1}{n-1}\sum_{i=1}^n \left(X_i-\bar X\right)^2 $ as the estimate of $\sigma$. Earlier we said that $ \frac{\bar X-\mu}{\sigma/\sqrt{n}} $ has a normal distribution with mean $0$ and variance $1$. The quantity $ \frac{\bar X-\mu}{S/\sqrt{n}}\tag{2} $ also has a distribution that does not depend on $\mu$ or $\sigma$. This is Student's t-distribution, introduced by the pseudonymous writer "Student", who was actually William Sealey Gossett. Thus we can find $A$ such that $(2)$ is in the interval bounded by $\pm A$ with probability $0.95$. It is a bigger number than $1.96$; formerly one looked it up in a table; now one uses software (it depends not only on the "$0.95$" but also on the sample size $n$). We get $ \bar X \pm A\frac{S}{\sqrt{n}} $ as endpoints of a confidence interval.
Similarly one can find a confidence interval for $\sigma^2$ by observing that the distribution of $ \frac{(n-1)S^2}{\sigma^2} $ does not depend on the unobservables $\mu$ and $\sigma$; this is a chi-square distribution. From $ A<\frac{(n-1)S^2}{\sigma^2} we get $ \frac{B}{(n-1)S^2} < \sigma^2 < \frac{A}{(n-1)S^2} $ and that is a confidence interval.
Many other examples exist.
-
0In this last example, Fisher's technique of conditioning on an ancillary statistic, where the latter is just the distance between the two observations, gives a reasonable answer. The ancillary statistic seems to take into account information that would be disregarded by reposing $50\%$ confidence in the $50\%$ confidence interval. But "information" in this instance doesn't mean what is usually called Fisher information, nor information in the sense of Fisher's concept of sufficiency. Just what it does mean would bear examination. I don't have an answer to that one. – 2012-09-09
Michael Hardy: I don't think anyone could give a better description of what the definition of a confidence interval is, so I will not try to give one of my own. But what I can add to the conversation is the fact that they are not unique and they are not always exact. There are exact confidence intervals and there are asymptotic ones.
Take the success parameter for a Bernoulli trial. If we have n iid Bernoulli random variables we can construct an exact confidence interval using the Clopper-Pearson method for evaluating cumulative binomial probabilities. There are also normal approximations to the binomial and so approximate confidence intervals can be constructed using normal approximations. When the sample size is large these approximate interval will have close to the advertised coverage. Coverage is the actual probability that if you repeat the procedure the new interval will contain the parameter (putting it in Michael Hardy's terms).
Since there can be more than one confidence interval for a parameter based on a random sample, how do we determine which one to use. Efron calls confidence intervals accurate if the actual coverage is equal to or close to the advertised coverage. Accuracy is a property we want every confidence interval to have if we would want to use it. The exact and asymptotic binomial confidence intervals are accurate. A confidence interval would be correct (term due to Efron) if among all accurate intervals its expected length is the smallest.