Entropy in a particular example

Question

I am new to this area of maths and am not feeling confident with my answers, I was wondering if someone could go through them with me and tell me if I'm right or where I've gone wrong.

Let $Z=\{(x,y)\in\mathbb{Z}\times\mathbb{Z} | 0\leq x\leq3, 0\leq y \leq10, y\leq x^2$}, and let Z be the discrete random variable that is uniform on the set of outcomes $Z$. Let X and Y be the discrete random variables representing the $x$ and $y$ coordinates of Z respectively.

a) Determine H(Y)

b) Determine H(Z|Y)

My attempts:

a) I know that H(Y) is given by $-\sum_{i=1}^np_i\log(p_i)$ (with log base 2) So would I get $-10\sum \frac{1}{10}\log(\frac{1}{10})$ = $3.322$?

b) I know H(Z|Y) is $-\sum_{y\in Y}\sum_{z\in Z}Pr(y)Pr(z|y)\log(Pr(z|y))$ and I got $-\sum_{y\in Y}\sum_{z\in Z}\frac{1}{10}\cdot \frac{1}{4}\log(\frac{1}{4}) = 0.05$ Which I know can't be right as it does not make sense to have such a low entropy.

Answer 1

Although $Z$ is uniformly distributed, neither $X$ nor $Y$ are.   Examine what points are in the sample space.

$\newcommand{\P}{\operatorname {\sf P}}\newcommand{\H}{\operatorname {\sf H}}\begin{align}Z~&=~\{(x,y)\in \Bbb Z^2: 0\leq x\leq 3, 0\leq y\leq \min\{10,x^2\}\}\\[1ex] & =~{\{(0,0), (1,0), (1,1), (2,0), (2,1),(2,2), (2,3), (2,4) \\(3,0), (3,1), (3,2), (3,3), (3,4), (3,5), (3,6), (3,7), (3,8), (3,9) \}} \\[1ex] &=~{\{(0,0),(1,0),(2,0),(3,0),\,(1,1),(2,1),(3,1),\\(3,2),(2,3),(3,3),(2,4),\,(3,4),(3,5),(3,6),(3,7),(3,8),(3,9)\}} \\[2ex] \P(Y{=}y) ~&=~ \begin{cases}\tfrac 4{18} & :& y=0\\ \tfrac 3{18} &:& y=1 \\ \tfrac 2{18} & :& y\in\{2,3,4\} \\ \tfrac 1{18} &:& y\in\{5,6,7,8,9\} \\ 0 &:& \text{elsewhere} \end{cases} \\[2ex] \P(Z{=}(x,y)\mid Y{=}y) ~&=~ \begin{cases}\tfrac 1 4&:& y=0, x\in\{0,1,2,3\}\\ \tfrac 13 &:& y=1, x\in\{1,2,3\} \\ \tfrac 12 &:& y\in\{2,3,4\}, x\in\{2,3\} \\ 1 &:& y\in\{5,6,7,8,9\}, x\in\{3\} \\ 0 &:& \text{elsewhere}\end{cases}\\[3ex] \H(Y)& = -\sum_{y=0}^9 \P(Y{=}y)\log_2\P(Y{=}y)\\[1ex] &= \tfrac 4{18}\log_2(\tfrac{18}4)+\tfrac 3{18}\log_2(\tfrac{18}3)+3\cdot\tfrac 2{18}\log_2(\tfrac {18}2)+5\cdot\tfrac 1{18}\log_2(18)\\[1ex] & =\dfrac{4+33\log_2(3)}{18} \\[1ex] &\approx 3.13 \\[2ex] \H(Z\mid Y) &= -\sum_{y=0}^9\P(Y{=}y)\sum_{x=0}^3 \P(Z{=}(x,y)\mid Y{=}y)\log_2\P(Z{=}(x,y)\mid Y{=}y) \\[1ex] &= -\sum_{x=0}^3\sum_{y=0}^9 \P(Z{=}(x,y))\log_2\P(Z{=}(x,y)\mid Y{=}y) \\ & = \tfrac 1{18}\big(4\log_2(4)+3\log_2(3)+6\log_2(2)+5\log_2(1)\big) \\[1ex] &= \frac{14+3\log_2(3)}{18} \\[1ex] &\approx 1.04\end{align}$

Answer 2

Your answer to a) cannot be correct since $y\leq x^2$ produces nonuniform $Y$. Besides entropy, this question is more about deriving distribution of $Y$.

What is it? $Z$ is a r.v. that takes on following values (uniformly): $(0,0)$, $(1,0)$, $(1,1)$, $(2,0)$, $\ldots$, $(2,4)$, $(3,0)$, $\ldots$, $(3,9)$, each with probability $\frac{1}{18}$. Out of the $18$ cases, $y=0$ in $4$, $y=1$ in $3$, $y\in\{2,3,4\}$ in $2$ and $y\in\{5,6,7,8,9\}$ in $1$. From this, you can calculate the entropy needed in a) and in b).