I find only the expression "this Gaussian is singular" on page 14 of your reference, but not the definition of "singular distribution".
But to answer your question:
The delta distribution is not a singular distribution, it is a discrete probability distribution. It does not have a Radon-Nikodym density with respect to the Lesbegue measure, because the Lesbegue measure of a single point is zero, and the delta distribution is concentrated on a single point.
Don't get confused if people write stuff like
$$
\int_{\mathbb{R}} \delta_0(x) d x = 1
$$
This is not correct in the strict sense. Instead, the "density function" of the delta distribution concentrated on zero - which is not a density in the sense of Radon-Nikodym - would be
$$
f(x) = 0 \; \text{for} \; x \neq 0
$$
and
$$
f(0) = \infty
$$
and therefore we would have
$$
\int_{\mathbb{R}} f(x) d x = 0
$$
But: For a discrete probability distribution, it is possible to name an at most countable set of points such that each point can be assigned a finite probability, such that the probability of any set is equal to the sum of the probabilities of the points it does contain.
This is not possible for a singular probability distribution like the Cantor distribution. The Cantor distribution is not concentrated on a countable set of points. Therefore the terms "singular distribution" and "discrete probability distribution" are different, and the delta distribution is a discrete one, not a singular one.