3
$\begingroup$

I've always used Shannon's entropy for measuring uncertainty, but I wonder why to use a logarithmic approach. Why shouldn't uncertainty be linear?

For instance, consider the following pairs of distributions:

$ \left[0.9, 0.1\right], \left[0.99, 0.01\right] $

$ \left[0.4, 0.6\right], \left[0.49, 0.51\right] $

Then you have the following uncertainty measures: $ H([0.9, 0.1]) = 0.46899559358928122\\ H([0.99, 0.01]) = 0.080793135895911181\\ H([0.4,0.6]) = 0.97095059445466858\\ H([0.49, 0.51]) = 0.9997114417528099 $

With the following example I just want show that it doesn't satisfy linearity:

$ H([0.9, 0.1]) - H([0.99, 0.01]) \simeq 0.3882 $

$ H([0.49, 0.51]) - H([0.4, 0.6]) \simeq 0.02876 $

As we can see, distributions that are closer to $[1,0]$ or $[0,1]$ tend faster to zero.

May be this is more a philosophic question but I think that may be someone could give me alternative measures of uncertainty that may be linear or, at least, provide some explanation to the rationale of this approach.

Thanks!

EDIT I don't mean linearity in the whole space but in the intervals $[0,\frac{1}{2}]$ and $[\frac{1}{2}, 1]$. Since, as @r.e.s. comments, is a required property for such a measure that $f(0) and $f(1)

  • 0
    let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/4365/discussion-between-ashok-and-kits89)2012-08-02

1 Answers 1

3

A linear function would not have some essential properties reasonably required of such a measure of uncertainty. Consider, as in your examples, distributions $[p, 1-p]$, with $0 \le p \le 1$, on a two-point sample space, and let $h(p)$ be the measure of uncertainty embodied by such a distribution. Among the basic requirements of such a measure are that both $h(0) < h(1/2)$ and $h(1) < h(1/2)$ -- because a probability $p$ equal to $0$ or $1$ corresponds to less uncertainty than does any other value of $p$ -- but this is impossible if $h(p)$ is linear in $p$, or indeed if $h(p)$ is monotonic in $p$. Standard axiomatic developments lead to the concave function $h(p) = - p \log p - (1-p) \log (1-p)$.

  • 0
    So, you could imagine the graph of the function that I propose similar to $-|x|$.2012-08-02