11
$\begingroup$

The Kullback-Leibler Divergence is defined as $$K(f:g) = \int \left(\log \frac{f(x)}{g(x)} \right) \ dF(x)$$

It measures the distance between two distributions $f$ and $g$. Why would this be better than the Euclidean distance in some situations?

  • 1
    Because $K(f\mid g)$ measures the ratio between the (un)likelihood that a $g$ sample is like an $f$ sample, and its typical likelihood as a $g$ sample.2011-12-11
  • 4
    There is an interpretation in terms of information theory, see http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Motivation.2011-12-11

2 Answers 2