2
$\begingroup$

The following is from a textbook one bayesian stats. that I can't understand some deduction. It is relevant about multiple parameters to be estimated.

The jth observation in the ith group is denoted by $y_{ij}$,
where

$$(y_{ij}|\mu_i,\sigma)\sim N(\mu_i,\sigma^2) \quad j=1,2, \dots, n_i \quad i= 1,2, \dots, m$$

Also the $y_{ij}$ are independent from each other.

Suppose $\mu_i \sim N(\mu,\tau^2)$ and denote
$$\theta= (\mu, \log(\sigma),\log(\tau))$$ $$Y=\{y_{ij}: j=1,\dots, n_i, i=1,\dots, n\}$$ $$Z=(\mu_1,\dots, \mu_m)$$ $$n=n_1+n_2+\cdots +n_m$$

So $\theta$ is the unknown parameters interested. Take its prior distribution as $p(\theta) \propto \tau$. Then by Bayes rule, it is not difficult to get the posterior distribution:

$$p(Z,\theta|Y) \propto p(\theta) \prod\limits_{i = 1}^m {p(\mu_i|\mu,\tau)} \prod\limits_{i = 1}^m \prod\limits_{j = 1}^{n_i} {p(y_{ij}|\mu_i,\sigma)}$$

This is the place I can't understand. How to get this formula if No.3 formula is not correct in this thread: I am confused about Bayes' rule in MCMC

Could someone explain it in detail? If there are any excellent books that could help me, please list them.

  • 1
    You can use $\TeX$ on this site by enclosing formulas in dollar signs; single dollar signs for inline formulas and double dollar signs for displayed equations. You can see the source code for any math formatting you see on this site by right-clicking on it and selecting "Show Math As:TeX Commands". [Here](http://meta.math.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference)'s a basic tutorial and quick reference. There's an "edit" link under the question.2012-10-12
  • 0
    Thanks for your help. I have modified it just now.2012-10-12
  • 0
    There are commands for all the Greek letters, e.g. `\sigma` yields $\sigma$.2012-10-12
  • 0
    I have tidied up your post - I don't think I've introduced any errors, but you should check.2012-10-12
  • 0
    Thanks very much. I forget the Greek letters.2012-10-12

1 Answers 1

2

AFAICT, the "trick" is that, by definition, $y_{ij}$ depends on $\mu$ and $\tau$ only via $\mu_i$. Thus, $$p(y_{ij}\mid\mu_i,\sigma) = p(y_{ij}\mid\mu_i,\sigma,\mu,\tau) = p(y_{ij}\mid\mu_i,\theta).$$

Similarly, $\mu_i$ does not depend on $\sigma$, so $$p(\mu_i\mid\mu,\tau) = p(\mu_i\mid\mu,\tau,\sigma) = p(\mu_i\mid\theta).$$ In particular, this means that we can rewrite your equation as

$$ \begin{aligned} p(Z,\theta\mid Y) \propto& p(\theta) \prod_{i = 1}^m p(\mu_i\mid\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}\mid\mu_i,\sigma) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i\mid\sigma,\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}\mid\mu_i,\sigma,\mu,\tau) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i\mid\theta) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}\mid\mu_i,\theta) \\ =& p(\theta)\, p(Z\mid\theta)\, p(Y\mid Z,\theta) \\ =& p(Y,Z,\theta) \\ =& p(Z,\theta\mid Y)\, p(Y). \end{aligned} $$

  • 0
    The reason $p(y_{ij}|\mu_i,\sigma) = p(y_{ij}|\mu_i,\sigma,\mu,\tau)$ is if $\mu_i$ is fixed in condition, then $\mu$ and $\tau$ are redundant due to the distribution of ${y_{ij}}$ is known with fixed $\mu_i$ and $\sigma$. Am I right to understand your point? There are so many places using this "trick". Thanks very much.2012-10-12
  • 0
    I have the following question for your deduction. How to get $\prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\theta) =p(Y|Z,\theta)$? Is there the equality that $p(X1|Y)*p(X2|Z)=p(X1,X2|Y,Z)$2012-10-12
  • 0
    For you first question, I _think_ I agree, if I understand you correctly. What really matters is that $y_{ij}$, by definition, depends only on $\mu_i$ and $\sigma$, and is therefore [conditionally independent](http://en.wikipedia.org/wiki/Conditional_independence) of anything else. As for your second question, indeed, $p(X_1|Y)\,p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\, Y$ and $X_2\perp Y\,|\,Z$, in which case $p(X_1|Y)=p(X_1|Y,Z)$ and $p(X_2|Z)=p(X_2|Y,Z)$. In particular, here, we know that, by definition, $y_{ij}\perp\mu_k$ whenever $i\ne k$.2012-10-12
  • 0
    Your answer helps. I appreciate it very much. I regret that my reputation is not enough to support your answer. But I will come back after accumulation is done. Another question: how to deduct $p(X_1|Y)p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z|Y$ and $X_2\perp Y|Z$? I know the meaning of $X_1\perp X_2$ but donot know the conditional form.2012-10-12
  • 0
    By definition, $A\perp B\,|\,C$ iff $p(A,B|C)=p(A|C)\,p(B|C)$. Divide both sides by $p(B|C)$ to get $p(A|B,C)=p(A|C)$. Apply with appropriate substitutions to get $p(X_1|Y)\,p(X_2|Z)=p(X_1|Y,Z)\,p(X_2|Y,Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\,Y$ and $X_2\perp Y\,|\,Z$.2012-10-12
  • 0
    Could I consider $A \perp B|C$ to be $(A \perp B)|C$, not $A \perp (B|C)$ ? Then it means A and B is conditional independent, seen from its equivalent statement $p(A,B|C)=p(A|C)p(B|C)$ which looks like $p(A,B)=p(A)p(B)$. I thought $\perp$ denotes uncorrelated in Prob.&Stats.2012-10-12
  • 0
    Yes, $A\perp B \,|\, C$ means "given $C$, $A$ is independent of $B$". You should _not_ read it as "$A \perp (B|C)$", whatever that might mean.2012-10-12
  • 0
    Thank you very much. I will remember the "trick" and other things you taught me. It has confused me for a long time.2012-10-12
  • 0
    @IlmariKaronen - More explicitly, sufficient conditions for $\ \ p(X_1,X_2|Y,Z)=p(X_1|Y)p(X_2|Z)\ \ $ are $\ \ (X_1\perp Z)\mid Y,$ $\ \ (X_2\perp Y)\mid Z,\ \ $ *and* $\ \ (X_1\perp X_2)\mid (Y,Z).\ \ $ (The latter condition was tacitly assumed in the above comments.)2017-11-24