1
$\begingroup$

I have to find the channel capacity of the following channel:

enter image description here

Now C = H(Y) - H(Y|X) (H() is the entropy-function). H(Y) = log2(2) = 1 at maximum and $$H(Y|X) = P(X=0)*p(a|X=0)log(1/p(a|X=0)) + P(X=1)*(p(a|X=1)log(1/p(a|X=1)) + p(b|X=1)log(1/p(b|X=1)))$$

This gives me: C = 1 - (P(X = 0) * 0 + P(X = 1) * 1), so the channel capacity is completely dependent on the distribution of X (the input). But intuitively I think that the channel capacity should be zero, because only one bit can be sent. But this doesn't correspond with the found formula. What did I do wrong?

  • 0
    Check out Z channel https://en.wikipedia.org/wiki/Z-channel_(information_theory) . Due to asymmetry, Uniform distribution is not going to achieve capacity. What bit do you think is favored by this channel? – 2017-01-30
  • 0
    @Gautam Shenoy Thanks, the 0-bit is favored and I've made some calculations. Now I found that the ideal input distribution is 2/3 chance of 0 bit and 1/3 chance of 1 bit. This gives a channel capacity of 0.585. Can this be correct? I found it be maximising the expression -p*log(p) - (1-p)*log(1-p) - p – 2017-01-30

1 Answers 1

1

There are several ways to solve this problem. Here is my alternate way:

Observe that the channel can be modelled as an AND gate. Namely $$ Y=X.Z$$ Let $Z$ be independent of $X$ and be distributed bernoulli $p$ where $p$ is the parameter of the Z channel (in this case $Pr(Y=1|X=1)=P(Z=1)=p$). In your case $p=1/2$.

In simple terms, when $X$ is $0$, the output is $0$ regardless of $Z$ but when $X$ is $1$, then output is equal to $Z$. Let $X$ be bernoulli $q$.

Then \begin{align} H(Y|X) &= (1-q)H(Y|X=0) + q H(Y|X=1) \\ &= qH(Z)= qh(p) \end{align} where $h(p)= -p\log p - (1-p)\log (1-p)$ is the binary entropy function. Now to calculate $H(Y)$, it suffices to calculate $Pr(Y=1)$. But $$Pr(Y=1) = Pr(X.Z=1)=Pr(X=1,Z=1)=Pr(X=1)P(Z=1)=pq $$

Hence $I(X;Y) = h(pq) - qh(p)$. For $p=1/2$, it equals $h(q/2)-q$. Now the derivative of $h(u)$ is $\log(\frac{1-u}{u})$. Hence we can show maxima is attained at $$q=2/5$$

You've made an error in the formula (in the comments) as this differs from what you have written. You took $h(q)-q$ instead of $h(q/2)-q$.

Note: This idea of modeling the channel as an AND gate may not appear very illuminating but when you chain multiple Z channels together, it helps a lot in the analysis.