This is an old contest problem, I wish I could remember where I first saw it. Anyway, André's comment that $b^4+4=(b^2-2b+2)(b^2+2b+2)$ is the key to a solution.
Looking modulo $16$, we see that $b^4+4$ cannot be a power of $2$. For $b>1 $, both factors will be strictly greater then $1$, so that if $p^k|(b^4+4)$ then $p$ must divide both $b^2-2b+2$, and $b^2+2b+2$. Since $\gcd(b^2-2b+2,b^2+2b+2)$ must divide $4b$, and $p$ divides both terms, we see that $p|b$. This then implies that $p$ divides $4$ which is impossible.
If $b=1$, then we get the one special case.