For $p=2$, the result is not true: taking $p=2$, $a=3$, $e^2$, we have that $x^2 \equiv 3\pmod{2}$ has a solution (any odd integer), but $x^2\equiv 3\pmod{2^2}$ has no solutions.
If $\gcd(a,p)=p$ the result is also not necessarily true: take $a=p$; then $x^2\equiv p\pmod{p}$ has a solution, but $x^2\equiv p\pmod{p^2}$ does not, since $x$ would have to be a multiple of $p$, and hence $x^2\equiv 0\pmod{p^2}$.
If $a$ is restricted to lying in $\{0,1,\ldots,p-1\}$, then these two conditions don't matter: for $p=2$, it is clear that both $x^2\equiv 0\pmod{2^e}$ and $x^2\equiv 1\pmod{2^e}$ have solutions for all $e\gt 0$; and if $p$ is odd and $a=0$, then $x^2\equiv 0\pmod{p^e}$ has solutions for all $e\gt 0$.
So, in any case, we can restrict to the case where $p$ is odd and $\gcd(a,p)=1$. In particular, any solution to $x^2\equiv a\pmod{p}$ or to $x^2\equiv a \pmod{p^e}$ must be relatively prime to $p$.
For odd primes, the problem can be solved using Hensel's Lemma, but one does not actually need it; just pushing it through what you are trying to do will do it, if you figure out what you need out of $k$ for things to work out.
Suppose $b^2 \equiv a \pmod{p^r}$, and you want to find $k$ such that $(b+kp^r)^2\equiv a \pmod{p^{r+1}}$.
Doing simple squaring, you have $$b^2 + 2bkp^r + k^2p^{2r}\equiv b^2 +2bkp^r \pmod{p^{r+1}}.$$ Now, $b^2 = a + tp^r$ for some $t$, so we want $$tp^r + 2bkp^r = p^r(t+2bk)\equiv 0 \pmod{p^{r+1}}.$$ This is equivalent to asking that $$t + 2bk\equiv 0 \pmod{p}.$$
So pick $k$ with $k(2b) \equiv -t\pmod{p}$ (which can be done because both $b$ and $2$ are relatively prime to $p$), and we are done.
By the way: one way to think of Hensel's Lemma is that it is the modular version of Newton's Method for approximating roots.
In Newton's Method, if $f'(b)\neq 0$, then you can go from $b$ to $b - \frac{f(b)}{f'(b)}$ as the "next approximation". Hensel's Lemma works the same way: you need $f'(b)$ to not be zero modulo $p$. Here we are working with $f(x) = x^2-a$; as long as $p\neq 2$, the formal derivative is not identically zero, which suggests what to do.
Notice the similarity with Newton's method in what we did: if $f(x) = x^2 - a$, then $f'(x)=2x$, so $f'(b)=2b$, and $f(b) = b^2-a = tp^r$, so $$ b - \frac{f(b)}{f'(b)} = b - \frac{tp^r}{2b} = b + \left(\frac{-t}{2b}\right)p^r$$ and what we are going to do is take $b+kp^r$ with $k$ given by $$k(2b)\equiv -t\pmod{p};$$ that is, $k$ is congruent to $\frac{-t}{2b}$ modulo $p$; precisely $-\frac{f(b)}{f'(b)}$ modulo $p$.