1
$\begingroup$

From Wasserman, All of Statistics, Theorem 9.6:

Let $\hat{\theta}_n$ denote the method of moments estimator. Under appropriate conditions on the model, the following statements hold:

  1. The estimate $\hat{\theta}_n$ exists with probability tending to $1$.

What does this precisely mean? Searching has not helped me find the precise meaning of this statement.

Additional context: $\hat{\theta}_n$ is calculated according to the method of moments to estimate $\theta$. So, for instance, if a probability distribution has one parameter and we have an iid sample $X_1, \dots, X_n$ following such a probability distribution, we would set $\mathbb{E}[X] = \bar{X}$, $\bar{X}$ being the arithmetic average of $X_1, \dots, X_n$. $\mathbb{E}[X]$ would likely be dependent on $\theta$; such a $\theta$ which solves the above equation is $\hat{\theta}_n$. With $k$ parameters $\theta_1, \dots, \theta_k$, we set $\mathbb{E}[X^j] = \dfrac{1}{n}\sum_{i=1}^{n}X_i^{j}$ for $j = 1, \dots, k$.

3 Answers 3

1

Ah, your additional context clarified matters.

Consider the space of all sequences $S$ of randoms drawn (each member of the sequence independently) from the distribution of interest. Let $\hat{\theta}_n[S]$ be the estimator based on the first $n$ elements of $S$. Now for most distributions we ordinarily work with, there is some number $\theta$ such that the set of sequences for which $\lim_{n\to\infty} \hat{\theta}_n[S] \neq \theta$ has measure zero. So $\lim_{n\to\infty} \hat{\theta}_n[S]$ "Almost Surely" exists and is equal to that $\theta$. (There are some long-tailed distributions for which this property does not hold.)

While the limit Almost Surely is $\theta$, the limit for any given sequence need not be $\theta$.

For example, for the unit normal distribution, $\lim_{n\to\infty} \hat{\theta}_n[S] = 0$ a.s. But imagine the sequence of randoms $\{1,1,1,1,\ldots\}$. This sequence is possible (although the probability of the union of all such sequences is vanishingly small). And that sequence will have the wrong limit for the estimator. In fact, for the sequence $\{1,4,9,16,\ldots\}$, the limit $\lim_{n\to\infty} \hat{\theta}_n[S]$ doesn't exist.

Nonetheless, if the "appropriate conditions on the model" hold, the method of estimation is useful because the limit "exists with probability one."

  • 0
    I didn't realize it did, thank you! So this looks like something that will be studied in greater depth once I get to the measure-theoretic treatment... – 2017-01-15
1

You have a probability $p_n$ that $\hat{\theta_n}$ exists, and $\lim_{n\to +\infty} p_n =1$

  • 0
    I don't think this is right. $n$ can't mean the number of points in the sample, because the estimator exists for any finite set of samples. Perhaps $n$ refers to the moment being estimated? – 2017-01-13
  • 0
    @MarkFischler Additional context provided. – 2017-01-13
1

Additional context would be helpful, but I assume that there's a chance that $\hat\theta_n$ isn't well defined in whatever set-up you have. Then it means that if we let $p_n$ be the probability that $\hat\theta_n$ exists, $$\lim_{n\to\infty}p_n=1$$

  • 1
    $\hat{\theta}_n$ is calculated according to the method of moments to estimate $\theta$. So, for instance, if a probability distribution has one parameter and we have an iid sample $X_1, \dots, X_n$ following such a probability distribution, we would set $\mathbb{E}[X] = \bar{X}$, $\bar{X}$ being the arithmetic average of $X_1, \dots, X_n$. $\mathbb{E}[X]$ would likely be dependent on $\theta$; such a $\theta$ which solves the above equation is $\hat{\theta}_n$. With $k$ parameters $\theta_1, \dots, \theta_k$, we set $\mathbb{E}[X^j] = \dfrac{1}{n}\sum_{i=1}^{n}X_i^{j}$ for $j = 1, \dots, k$. – 2017-01-13