This is wrong. In what follows, we assume that we are working with the operator norm.
If $A$ is non-singular, it is easy to check that $P$ is non-singular for all $P\in B\left(A;\frac{1}{\Vert A^{-1}\Vert}\right)$.
The proof runs as follows. Let $P\in B\left(A,\frac{1}{\|A^{-1}\|}\right)$. We
shall prove that $P\in GL_n(\mathbb{R})$. For each $x\in\mathbb{R}^n$,
\begin{eqnarray*}
\|x\|&=&\|A^{-1}(Ax)\|\leqslant \|A^{-1}\|\|Ax\|\\& \leqslant &
\|A^{-1}\|\left(\|(A-P)x\|+\|Px\|\right)\leqslant
\|A^{-1}\|\left(\|A-P\|\|x\|+\|Px\|\right).
\end{eqnarray*}
Thus,
$$
\|x\|\leqslant \frac{\|A^{-1}\|}{1-\|A-P\|\|A^{-1}\|}\|Px\|,\text{
for all }x\in\mathbb{R}^n.
$$
This implies that $P\in GL_n(\mathbb{R})$ because if $ Px=0$, we have
$x=0$. Thus we have proved that $
B\left(A,\frac{1}{\|A^{-1}\|}\right)\subset GL_n(\mathbb{R})$.