2
$\begingroup$

If a PDF is re-constructed as following $p(x)=\sum_{k=1}^{K}\pi_k p(x|k),$ where $p(x|k)$ has mean $\mu_k$ and covariance matrix $\Sigma_k$ , compute the mean and covariance of the new mixture distribution.

Finding the mean is trivial , so say that I've found the mean and denote it as $mean=\left[ \begin{array}{cl} mean_{1} \\ ... \\ mean_{D} \end{array}\right]$

I'm quite confused about the dimension of covariance matrix, say I have $D$ dimension for each observation $x=\left[ \begin{array}{cl} x_{1} \\ ... \\ x_{D} \end{array}\right]$, then dimension of covariance matris should be $D \times D,$ but what 's the concrete element in position $$ ?

1 Answers 1

4

I think what you mean is this. There is a random variable $Y$ with possible values $1, \ldots, K$ and corresponding probabilities $\pi_k$. Your random variable $X$ is a $d$-dimensional vector with conditional expectations $\text{E}[X | Y=k] = \mu_k$ (a $d$-dimensional vector) and conditional covariances $\text{Cov}(X | Y=k) = C_k$ (a $d \times d$ matrix). I'd rather use this instead of $\Sigma_k$ because I want to use $\sum_k$ for summation over $k$. Thus $\text{Cov}(X_i,X_j | Y = k) = \text{E}(X_i X_j | Y = k) - \text{E}(X_i | Y=k) \text{E}(X_j | Y = k) = (C_k)_{ij}$. Now the unconditional expectation is $\text{E}(X) = \sum_k E(X| Y=k) \pi_k = \sum_k \mu_k \pi_k$. On the other hand, $\text{E}(X_i X_j) = \sum_k \text{E}(X_i X_j | Y=k) \pi_k = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j\right) \pi_k$ so $ \eqalign{\text{Cov}(X_i,X_j) &= \text{E}(X_i X_j) - E(X_i) E(X_j)\cr & = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j \right) \pi_k - \sum_k (\mu_k)_i \pi_k \sum_{k'} (\mu_k')_j \pi_{k'}\cr} $ which can be expressed as $ \text{Cov}(X_i,X_j) = \text{E}\left[{\text{Cov}(X_i,X_j | Y)}\right] + \text{Cov}(E(X_i|Y), E(X_j | Y))$ the "law of total covariance" formula: see http://en.wikipedia.org/wiki/Law_of_total_covariance These entries form a matrix $ C = \sum_k C_k \pi_k + \sum_k \mu_k^T \mu_k \pi_k - \mu^T \mu $ where $\mu = E(X)$.

  • 0
    Thanks for you answer buddy, but I can't tell if this is right or wrong because I actually got the same reasoning, as my exams are just finished yesterday, I'll consult my professor for this problem soon, hope he will give me the same result :)2012-05-26