Assume $G$ is a finite abelian $p$-group with a unique subgroup of order $p$. I claim that if $a,b\in G$, then either $a\in\langle b\rangle$ or $b\in\langle a\rangle$. This will show that $G$ is cyclic, by considering all elements of $G$ one at a time.
Let $a,b\in G$; by exchanging $a$ and $b$ if necessary, we may assume that $|a|\leq |b|$, and we aim to prove that $a\in\langle b\rangle$. If $|a|\leq p$, then $\langle a\rangle$ is contained in the unique subgroup of order $p$, and hence is contained in $\langle b\rangle$, and we are done. So say $|a|=p^k$, $|b|=p^{\ell}$, $1\lt k\leq \ell$.
Let $t$ be the smallest nonnegative integer such that $a^{p^t}\in\langle b\rangle$. Note that since $a^{p^{k-1}}$ and $b^{p^{\ell-1}}$ are both of order $p$, the fact that $G$ has a unique subgroup of order $p$ means that $\langle a^{p^{k-1}}\rangle = \langle b^{p^{\ell-1}}\rangle$, so $t\leq k-1$; that is, $a^{p^t}\neq 1$. And since $a^{p^{t}}$ is of order $p^{k-t}$, we must have $\langle b^{p^{\ell-k+t}}\rangle = \langle a^{p^{t}}\rangle$. Let $u$ be such that $a^{p^{t}} = b^{up^{\ell-k+t}}$.
Now consider $x=ab^{-up^{\ell-k}}$. Note that since $k\leq \ell$, this makes sense. What is the order of $x$? If $x^{p^r}=1$, then $a^{p^r} = b^{up^{\ell-k+r}}\in\langle b\rangle$, so $r\geq t$ by the minimality of $t$. And $x^{p^t} = a^{p^t}b^{-up^{\ell -k + t}} = b^{up^{\ell-k+t}}b^{-up^{\ell-k+t}} = 1.$ So $x$ is of order $p^t$.
If $t\gt 0$, then $x^{p^{t-1}}$ has order $p$, so $x^{p^{t-1}}\in \langle b\rangle$. But $x^{p^{t-1}} = a^{p^{t-1}}b^{-up^{\ell-k+t-1}},$ so the fact this lies in $\langle b\rangle$ means that $a^{p^{t-1}}\in\langle b\rangle$. The minimality of $t$ makes this impossible.
Therefore, $t=0$, which means $x=1$. Thus, $a^{p^0} = a\in\langle b\rangle$, as desired.
Added. The above argument shows that an abelian group $A$ in which every element has order a power of $p$ and that contains a unique subgroup of order $p$ is locally cyclic; that is, any finitely generated subgroup of $A$ is cyclic. This includes some groups that are not finite or finitely generated, e.g. the Prüfer $p$-groups.