1
$\begingroup$

I'm familiar with information and coding theory, and do know that the units of Shannon information content (-log_2(P(A))) are "bits". Where "bit" is a "binary digit", or a "storage device that has two stable states".

But, can someone rigorously prove that the units are actually "bits"? Or we should only accept it as a definition and then justify it with coding examples.

Thanks!

2 Answers 2

2

When there are two events, both of which are equally likely, when conveying the news that one of them has actually happened, you convey -log2(0.5)=1 bit of information.

There is no rigorous proof here: just a mapping from a probability space to a bit-space (if I may call so). Either you see a binary random variable with equally likely 0 or 1, or you consider two equally likely stable states, whose representation would entail 1 bit.

  • 0
    @Michael -log2(0.1) = 3.3222017-07-03
1

A quick answer could be that if your storage device has probability $1/2$ of being in either of the two possible states, then the formula $\sum_i -p_i\log_2(p_i)$ gives you $=1$, so that is the unit. Somehow it sounds like you are not satisfied with that argument given that you have surely seen it?? Note that this does require the two states to be equiprobable. Note also that some (older?) sources occasionally measure information in nats instead of bits. The difference being that the natural logarithm is used in place of base-2 logarithm.

  • 0
    Thanks Jyrki. Your example is good to demonstrate the relation betwee bits and probability of the events. But it just another example and not a proof. So, as I wrote in the above $c$omments, it looks like we shoul$d$ accept the units of "bits" as **definition** and then justify it using various examples.2012-03-28