Easiest to specify the squares among the units. Remember that, using standard $p$-ary notation, a $p$-adic unit is of form $\dots a_4a_3a_2a_1a_0;$ with $a_0\ne0$.
The situation is special for $p=2$, since it’s not enough for $a_0$ to be a square. Rather, the $2$-adic units that are squares are those that are $\equiv1\pmod8$, in other words the right-hand end must be $\dots a_4a_3001;$ (note that $3=11;$, $5=101;$, and $7=111;$ all are non-squares in $\Bbb Q_2$, but $-7=\dots1111001;$ is a square).
For primes bigger than $2$, what you say is (almost) right: for a unit, the $a_0$ must be a nonzero square modulo $p$.
For nonzero nonunits $z$, you may always write $z=p^ru$, with $r\in\Bbb Z$ and $u$ a $p$-adic unit. Then $z$ is a square if and only if $r$ is even and $u$ is a square unit.