13
$\begingroup$

How to prove the following known (Pinsker's) inequality?

For two strictly positive sequences $(p_i)^n_{i=l}$ and $(q_i)^n_{i=l}$ with $\sum_{i=1}^np_i=\sum_{i=1}^nq_i=1$ one has $\sum_{i=1}^np_i\log\frac{p_i}{q_i}\ge \frac{1}{2}\left(\sum_{i=1}^n|p_i-q_i|\right)^2.$

  • 1
    Check also Beck & Teboulle 2003, "Mirror descent and nonlinear projected subgradient methods for convex optimization", Proposition 5.1 for elementary proof of a weaker inequality with symmetrized KL-divergence.2015-09-15

2 Answers 2

6

See here. Pinsker's inequality

https://mathoverflow.net/questions/42667/two-reference-requests-pinskers-inequality-and-pontryagin-duality

  • 0
    @Kolmo can you please point out the condition for equality in Pinsker's inequality?2016-11-12
4

Take a look at Tsybakov 2009 Introduction to Nonparametric Estimation, p.88, also available in the net. Good and simple proof.