6
$\begingroup$

$X$ is a random variable with normal distribution, assume $Y=X$, what is the mutual information $I(X;Y)$?

I guess that $h(Y|X)=0$ since when $X$ is known, $Y$ is completely known, so $$I(X;Y)=h(Y)-h(Y|X)=h(Y)=\frac{1}{2}\log 2\pi e\sigma^2$$ nat.

But, I was told I was wrong! and a numerical computation also shows that the value of $$I(X;Y) \neq \frac{1}{2}\log 2\pi e\sigma^2$$ Where is my mistake? Please help me out of this problem, thanks a lot! (Please note that $X$ and $Y$ are both continuous).

2 Answers 2

0

If $(X_1,X_2)$ is a Gaussian vector each variable having the same variance ($\sigma^2$) with covariance matrix $K$, then $$h(X_1,X_2)=\frac{1}{2}\log((2\pi e)^2|K|)=\frac{1}{2}\log((2\pi e)^2 \sigma^4(1-\rho^2))$$ where $\rho$ is the correlation coefficient.

Now $I(X_1:X_2)=2h(X_1)-h(X_1,X_2)=-\frac{1}{2}\log(1-\rho^2)$.

When $X_1=X_2$, $\rho=1$ and hence $I(X:X)=\infty$.

  • 0
    so h(X|X) should be $-\infty$ instead of 0 as in discrete case. and if i run that program using more and more samples, the program will produce larger and larger outcome. Thank you!2012-06-20
  • 1
    This answer is **wrong**. It should be unselected. The answer by @avitus is the right one.2015-04-10
5

For any r.v. $X$

$$I(X,X)=H(X).$$

To see this, put $Y=X$ in $I(X,Y)=H(X)-H(X|Y)$ and use

$$H(X|X)=0.~~ (*)$$

In summary, the mutual information of $X$ with itself is just its self information $H(X)$, as $H(X|X)$, i.e. the residual average information carried by $X$ conditional $X$ is zero, i.e. $(*)$ holds.