5
$\begingroup$

From Wikipedia:

Let $X$ be a random variable with a probability density function $f$ whose support is a set $\mathbb{X}$. The differential entropy $h(f)$ is defined as $ h(f) = -\int_\mathbb{X} f(x)\log f(x)\,dx \quad \text{i.e.} \quad \mathrm{E}_f(-\log f(X)). $

The discrete entropy of a discrete probability measure is also defined as $\mathrm{E}_p (-\log p(X))$, where $p$ is the mass probability function, which can be viewed as the density function with respect to the counting measure on the discrete sample space.

Am I right that the concept of entropy depends on the underlying measure of the sample space, since the integrand is the density function wrt some underlying measure?

If yes, what is the underlying measure on the sample space for differential entropy that Wiki refers to in the above quote?

What is the/some general structure(s) defined on the sample space for it to be meaningful to discuss entropy, more than $\mathbb{R}^n$ with Lebesgue measure being the underlying measure?

Thanks and regards!

  • 0
    If I agreed with that, I wouldn't have asked.2012-01-06

1 Answers 1

3

In general, one can define the relative entropy between two probability measures $\mu$ and $\nu$ (given that $\nu\ll\mu$) as $H(\nu\mid\mu)=-\int\frac{d\nu}{d\mu}(x)\log\left(\frac{d\nu}{d\mu}(x)\right)\mu(dx)$ where $\frac{d\nu}{d\mu}$ is the Radon-Nikodym derivative of $\nu$ with respect to $\mu$. In fact, one can put any convex function $U:\mathbb{R}\to\mathbb{R}$ instead of $U(x)=-x\log(x)$.

When one speaks of a probability density, one is speaking of a real-valued r.v. $X$ whose law $\mu_X(A)=\mathbb{P}(X^{-1}(A))$ is absolutely continuous with respect to the Lebesgue measure $\lambda$ on $\mathbb{R}$ and the density function $f$ is nothing more than the Radon-Nikodym derivative $\frac{d\mu_X}{d\lambda}$. In the case of a discrete r.v. one replaces the Lebesge measure with the counting measure on $\mathbb{N}$.

With this in mind, is easy to see that the definition you gave above is nothing more than the entropy $H(\mu_X\mid\lambda)$, where $\mu_X$ is the law of $X$ and $\lambda$ is the Lebesgue measure on $\mathbb{R}$.