This is a great example of why one should always present the original problem and not just some intermediate stage of the solution. This is particularly relevant if you're already suspecting that you went wrong somewhere – why assume that you know which part is wrong and present the other part as given when you already think there must be a mistake somewhere?
In the present case you made two mistakes in deriving what you at first presented as the given problem. One seems to be just a problem of notation – it seems that you intended this to be summed over all integer $n$, but that's not what you wrote. The other is that you dropped a factor of $r^2$ in the argument of the second $\delta$. Here's a calculation of your original integral:
\begin{eqnarray} \int\limits_{-\infty}^\infty\int\limits_{-\infty}^\infty \phi(x^2+y^2)\delta'(x^2+y^2-a)\delta(x^2-y^2)\mathrm dx\mathrm dy &=& \int\limits_0^\infty\int\limits_0^{2\pi} \phi(r^2)\delta'(r^2-a)\delta(r^2\cos2\theta)\mathrm rd\theta\mathrm dr \\ &=& \frac12\int\limits_0^\infty\int\limits_0^{2\pi} \phi(u)\delta'(u-a)\delta(u\cos2\theta)\mathrm d\theta\mathrm du \\ &=& 2\int\limits_0^\infty\frac{\phi(u)}u\delta'(u-a)\mathrm du \\ &=& 2\int\limits_0^\infty\frac{\phi(u)-u\phi'(u)}{u^2}\delta(u-a)\mathrm du \\ &=& 2\frac{\phi(a)-a\phi'(a)}{a^2} \;. \end{eqnarray}
This is also not the result \phi(a)-\phi'(a) that you expected. Perhaps you're assuming $a=1$ somewhere? Then there would only be an excess factor of $2$.