Confused about the definition of a random sample, statistics and estimators/estimates

Question

I'm currently studying basic statistics, and I don't really understand the definition of random sample in the book I'm reading (Introduction to Probability and Statistics: Principles and Applications for Engineering and the Computing Sciences, 4th Edition).

The book defines random samples as follows:

[...] the term "random sample" is used in three different but closely related ways in applied statistics. It may refer to the objects selected for study, to the random variables associated with the objects to be selected, or to the numerical values assumed by those variables.

A random sample of size $n$ from the distribution of $X$ is a collection of $n$ independent random variables, each with the same distribution as $X$.

[...] The objects selected generate $n$ numbers $x_1,x_2,x_3,\ldots,x_n$ which are the observed values of the random variables $X_1,X_2,X_3,\ldots,X_n$.

At this point, I don't really see the point of thinking of a random sample as a collection of random variables, considering the data gathered for an experiment is a set of constants.

A statistic is defined by the book as "a random variable whose numerical value can be determined from a random sample". As an example, the sample mean statistic of a sample (a set of random variables) $\{1, 2, 3\}$ is defined as $$ \bar{X} = \frac{X_1 + \cdots + X_n}{n} $$ where $X_{1..n}$ are random variables (which means that $\bar{X}$ is also a random variable), as opposed to $$ \bar{x} = \frac{x_1 + \cdots + x_n}{n} $$ where $x_{1..n}$ are constant numbers (which means that $\bar{x}$ is a single number).

Now, since $X_1, \ldots, X_n$ are random variables but each have a single, known value ($X_i$ assumes the value $x_i$), $\bar{X}$ is also a random variable which assumes a known value -- i.e., $(x_1 + \cdots + x_n)/n$. (Please correct me if I'm wrong here.)

Here's my confusion. I think I've understood the concept of statistics and estimators -- to provide an estimation for population parameters using the characteristics of a sample. However, I don't see a reason behind thinking of a random sample as a set of random variables -- as opposed to the (in my eyes) more "natural" way of thinking of them as simply a set of numerical constants -- or thinking of a statistic like the sample mean, median, range, etc. as a random variable rather than a single, constant numerical value.

---

This is my first post here, so the question probably has problems with structure, length and clarity, but what I'm essentially asking is for some "justification" for thinking of a) a random sample as a set of random variables as opposed to a set of numbers, and b) a statistic as a random variable rather than a constant.

Answer 1

You need to remember that we don't usually take just one sample, but often many repeated samples from the same population.

Suppose we take a sample of size $2$ from some large population. The first time I might get $3$ and $7$. The second time I might get $9$ and $4$. The third time I might get $6$ and $3$. Each time I take a fresh sample, the two values are potentially different to what they were before. It is natural to think of these as being random variables because I know the properties of random variables.

If I also calculate the mean, I will have a different value for the "sample statistic" each time: first $5$, then $6.5$ and then $4.5$

Again, because the sample mean is changing each time, it is natural to treat the sample statistic as a random variable.

Answer 2

In a statistical model, you assume that the sample $x_1\ldots x_n$ is drawn from some distribution $f_{X_1\ldots X_n}(x_1,\ldots ,x_n)$ so the actual numerical values of the sample are a realization of the random variables $X_1\ldots X_n.$ This is a model. We are assuming that the sample is an outcome of a random process.

The reason this is useful, and also why considering sample statistics like the sample mean as random variables is useful, is that we can use the sample to fit and validate the model. So if we have a model in mind and we collect a sample, it is useful to know what the probability of that sample is under the model (this is called likelihood). If we draw a very improbable sample (in some sense), it will cast doubt on the validitity of the model. And if the model has free parameters, we can choose the parameters as ones that are consistent with our sample (e.g. maximum likelihood estimation).

For instance, take a simple artificial example where we believe that our sample is independent normals with known variance $1$. But we don't know the mean. To fit the mean from the data, we look at the sample mean $\bar x$ and estimate $\hat \mu = \bar x.$ It turns out that this provides the best estimate for the mean under the model. However, we'd like to know two other things

1. what is the error in our estimate.

2. is our model any good, anyway?

It turns out considering sample statistics as random variables is useful for both of these.

We usually assume 2 when considering 1 (otherwise, what would be the point). So for 1, we want to know the error of the estimate. Obviously if we have like 5 samples, our error will be pretty big, and will decrease if we have more samples. We can make this quantitative by looking at the distribution of the sample mean (as as random variable). If the true mean of the normals is $\mu$, then the sample mean of $n$ samples will have distribution $\mathcal{N}(\mu, \frac{1}{n}),$ i.e. will be normal with variance $1/n$ (standard deviation $1/\sqrt{n}$), This tells you that the sample mean from the sample you draw will probably be within about $1/\sqrt{n}$ of the true mean. This gives an immediate estimate of the error and you can get more quantitative if you wish (i.e. confidence intervals).

For 2, we could look at our sample and take the sample variance. If our sample is large, we'd expect this to come out to close to 1. If it doesn't, our model might be wrong. Say we from a sample of 100 we get a sample variance of 1.1. Does that mean our model is wrong? In order to get a quantitative handle on the question, it's useful to consider the distribution of the sample variance under our model. Then we could compute something like $Prob(\mbox{sample variance} > 1.1)$ under the model. If this is very low, we have cause to question the assumptions of our model. In this case, the natural assumption to question is that the $X_i$'s have variance 1.

We could then relax that assumption and fit a new model where the variance is unknown. Then if we are still skeptical of some of our assumptions, we could think of another statistic to test (say, 4th moment or autocorrelation) and do it the same as before. If these come out looking unlikely, maybe the assumption of normality or independence is wrong.