I think the question should be interpreted in this way: Given the hypothesis show that $P$ is contained in a subgroup of $\mathrm{Sym}(n)$ isomorphic to $\mathrm{Sym}(n-1)$. And this is absolutely clear: indeed it's true a little bit more, $P$ is contained in a subgroup of $\mathrm{Sym}(n)$ isomorphic to $\mathrm{Sym}(n-r)$, where $r$ is a nonnegative integer less then $p$, equivalent to $n \pmod p$. Indeed $P$ is a $p$-group acting on a set, so it has a set of fixed point of cardinality equivalent to $n \pmod p$ (orbit stabilizer+n=sum of the cardinality of orbits). Then clearly there are at least $r$ points fixed by the whole $P$. So $P$ has a natural immersion in the stabilizer of these $r$ points, that is isomorphic to $\mathrm{Sym}(n-r)$.