2
$\begingroup$

I have been reading a bit about conditional entropy, joint entropy, etc but I found this: $H(X|Y,Z)$ which seems to imply the entropy associated to $X$ given $Y$ and $Z$ (although I'm not sure how to describe it). Is it the amount of uncertainty of $X$ given that I know $Y$ and $Z$? Anyway, I'd like to know how to calculate it. I thought this expression means the following:

$$H(X|Y,Z) = -\sum p(x,y,z)log_{2}p(x|y,z)$$

and assuming that $p(x|y,z)$ means $\displaystyle \frac{p(x,y,z)}{p(y)p(z)}$, then \begin{align} p(x|y,z)&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}\frac{p(x,y)}{p(y)}\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}p(x|y) \\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}\frac{p(x,z)}{p(z)}p(x|y)\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}p(x|z)p(x|y) \end{align} but that doesn't really help.

Basically I wanted to get a nice identity such as $H(X|Y)=H(X,Y)-H(Y)$ for the case of two random variables.

Any help?

Thanks

2 Answers 2

4

$$H(X\mid Y,Z)=H(X,Y,Z)-H(Y,Z)=H(X,Y,Z)-H(Y\mid Z)-H(Z)$$ Edit: Since $\log p(x\mid y,z)=\log p(x,y,z)-\log p(y,z)$, $$ H(X\mid Y,Z)=-\sum\limits_{x,y,z}p(x,y,z)\log p(x,y,z)+\sum\limits_{y,z}\left(\sum\limits_{x}p(x,y,z)\right)\cdot\log p(y,z). $$ Each sum between parenthesis being $p(y,z)$, this proves the first identity above.

  • 0
    Can you show me how to manipulate $p(x|y,z)$ to get that identity? I can see why it's reasonable to get $H(X,Y,Z)-H(Y,Z)$ from $H(X|Y) = H(X,Y)-H(Y)$ but I'd like to know how to work out that from probability distributions.2012-07-08
  • 0
    See Edit. This uses nothing but the definition, really.2012-07-09
1

Yes, entropy is often referred to as "uncertainty", so $H(X|Y)$ can be thought of as your uncertainty about $X$, given that you know $Y$. If it's zero, then we would say that knowing $Y$ tells us "everything" about $X$, and so on.

It might be easier to think in terms of just two variables, although your basic idea is right. You can see wikipedia for more explicit calculations.

  • 0
    Thanks. Yes, I think I understand the conditional entropy correctly, however, I find it a bit awkward with two "conditional variables", though. What about my calculation? Unfortunately, Wikipedia didn't help a lot because it doesn't provide $H(X|Y,Z)$.2012-07-08
  • 0
    @Robert: As did said, you can use the [chain rule](http://en.wikipedia.org/wiki/Conditional_entropy#Chain_rule) on that wikipedia page to change $H(X|Y,Z)$ into an expression involving only $H(Y|Z)$2012-07-08