6
$\begingroup$

We know that $l_i=\log \frac{1}{p_i}$ is the solution to the Shannon's source compression problem: $\arg \min_{\{l_i\}} \sum p_i l_i$ where the minimization is over all possible code length assignments $\{l_i\}$ satisfying the Kraft inequality $\sum 2^{-l_i}\le 1$.

Also $H(p)=\log \frac{1}{p}$ is additive in the following sense. If $E$ and $F$ are two independent events with probabilities $p$ and $q$ respectively, then $H(pq)=H(p)+H(q)$.

As far as I know, mainly for these two reasons $H(p)=\log \frac{1}{p}$ is considered as a measure of information contained in a random event $E$ with probability $p>0$.

On the other hand, if we average the exponentiated lengths, $\sum p_i2^{tl_i}, t>0$, subject to the same Kraft inequality constraints, the optimal solution is $l_i=\log \frac{1}{p_i'}$ where $p_i'=\frac{p_i^{\alpha}}{\sum_k p_k^{\alpha}}, \alpha=\frac{1}{1+t}$, called Campbell's problem.

Now $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ is also additive in the sense that $H_{\alpha}(p_i p_j)=H_{\alpha}(p_i)+H_{\alpha}(p_j)$. Moreover $H_{\alpha}(1)=0$ as in the case of Shannon's measure.

Also note that, when $\alpha=1$, $H_1(p_i)=\log \frac{1}{p_i}$ we get back Shannon's measure.

My question is, are these reasons suffice to call $H_{\alpha}(p_i)=\log \frac{1}{p_i'}$ a (generalized) measure of information?

I don't know whether the dependence of measure of information of an event also on the probabilities of the other events make sense.

  • 1
    Your notation is a bit unconventional. Normally $H(p)$ is used for *Shannon entropy* of a probability distribution (and not for a single value "p") given by $H(p) = -\sum_ip_i\log p_i$. The additivity you've described is a deeper property of this entropy function, not just a property of the logarithm as you've mentioned. There are a number of reasons why $H(p) = -\sum_ip_i\log p_i$ is used as the entropy function and not anything else. Refer to http://www.maths.gla.ac.uk/~tl/Renyi.pdf for the details. Hope this helps.2011-12-26
  • 0
    @VSJ Thanks for the response. $H(p)$ is also used to denote just $\log(1/p)$. If you have a look at the book _On measures of information and their characterizations_ by Aczel and Daroczy, Chapter 0, section 2, you would notice that. It is proved in that section that $H(p)=\log(1/p)$ is the only function satisfying i) $H(p)\ge 0$, ii) $H(pq)=H(p)+H(q)$ and iii) $H(1/2)=1$. They say that for this reason $\log(1/p)$ is considered as a measure of information. My curiosity is whether $\log(1/p_i')$ can also be considered as a measure of information as it has the crucial additivity.2011-12-27
  • 0
    Also note that $\log(1/p_i')$ makes sense only when all the probabilities $p_1,\dots,p_n$ concerned with an experiment are considered. In that book they make a crucial assumption that _the measure of information of an event depends only upon the probability of the event considered_. But in the case of $\log(1/p_i')$, it depends probabilities of all events possible except when $\alpha=1$.2011-12-27
  • 0
    Aslo, please remember that I am not talking about the average here. My question is the following. An experiment is performed. $E_1,\dots,E_n$ are the possible disjoint events with respective probabilities $p_1,\dots,p_n$. Can we regard $\log(1/p_i')$ as a measure of information associated with $E_i$ as this answers a source compression problem and has the additivity property?2011-12-27
  • 0
    @Ashok I was wondering, I have never heard of Shannon's source compression problem. However, I am sure its a very interesting theory and wanted to learn more about it. I was wondering, if you knew a good source to learn about it? textbook, blog, lecture note, website, etc? Thanks in advance!2014-11-09
  • 0
    @Pinocchio: You can try to solve the optimization problem by yourself. It's really easy. If not a good reference is Cover and Thomas's Information theory book. Feel free to ask if any questions.2015-07-07

1 Answers 1

2

That's exactly the extension known as Renyi entropy with a normalization factor of $\frac{1}{1-\alpha}$.

$$H_\alpha(X) = \frac{1}{1-\alpha}\log\Bigg(\sum_{i=1}^n p_i^\alpha\Bigg)$$

  • Rényi, Alfréd (1961). "On measures of information and entropy". Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561.
  • 0
    //That's exactly the extension known as Renyi entropy with a normalization factor of...// Which one do you mean?2014-11-11