This is false.
In fact, every finitely generated group, simple or not, is the fundamental group of a compact 4-manifold $X$ with boundary $S^3$, a 3 dimensional sphere.
First, it is known that every finitely generated group $G$ is the fundamental group of a compact 4 dimensional manifold $M$. See for example https://mathoverflow.net/questions/15411/finite-generated-group-realized-as-fundamental-group-of-manifolds.
Now, let $B^4$ be a small open ball in $M$ and consider $X = M - B^4$. Then $X$ is a manifold with boundary $S^3$.
I claim that $\pi_1(X) = \pi_1(M) = G$.
To see this, write $M = X \cup B^4$ and apply Seifert-van Kampen. Using the fact that $X\cap B^4 = S^3$ is simply connected and that $B^4$ is simply connected, we learn that $\pi_1(M)\cong \pi_1(X)\ast_{\pi_1(S^3)} \pi_1(B^4) \cong \pi_1(X)$. Thus, $\pi_1(X) \cong \pi_1(M) = G$ as claimed.