It's clear that $f$ splits over any extension of $F$ in which it has a root. Suppose that it does not have a root in $F$, i.e., that $a$ is not a $p$-th power in $F$, and let $K$ be a field containing a root $\alpha$ of $f$, so $\alpha^p=a$. Let $g$ be a monic irreducible factor of $f$ in $F[x]$. Since $f(x)=(x-\alpha)^p$ in $K[x]$, we must have $g=(x-\alpha)^r$ for some $r$ between $1$ and $p$. In fact, since $\alpha\notin F$, $r>1$, and, assuming $f$ is not irreducible, we must have $\deg(g). Now, since $(x-\alpha)^r\in F[x]$, in particular, the coefficient of $x^{r-1}$ is in $F$. Using the binomial theorem, we see that this coefficient is $\pm r\alpha$. But because $r$ is prime to $p$, it is a unit in $F$, and we see that $\alpha=(\pm r)^{-1}(\pm r\alpha)\in F$. This is a contradiction. So we must have $g=f$, and $f$ is irreducible.