This is wrong. In what follows, we assume that we are working with the operator norm. If $A$ is non-singular, it is easy to check that $P$ is non-singular for all $P\in B\left(A;\frac{1}{\Vert A^{-1}\Vert}\right)$.
The proof runs as follows. Let $P\in B\left(A,\frac{1}{\|A^{-1}\|}\right)$. We shall prove that $P\in GL_n(\mathbb{R})$. For each $x\in\mathbb{R}^n$, \begin{eqnarray*} \|x\|&=&\|A^{-1}(Ax)\|\leqslant \|A^{-1}\|\|Ax\|\\& \leqslant & \|A^{-1}\|\left(\|(A-P)x\|+\|Px\|\right)\leqslant \|A^{-1}\|\left(\|A-P\|\|x\|+\|Px\|\right). \end{eqnarray*} Thus, $ \|x\|\leqslant \frac{\|A^{-1}\|}{1-\|A-P\|\|A^{-1}\|}\|Px\|,\text{ for all }x\in\mathbb{R}^n. $ This implies that $P\in GL_n(\mathbb{R})$ because if $ Px=0$, we have $x=0$. Thus we have proved that $ B\left(A,\frac{1}{\|A^{-1}\|}\right)\subset GL_n(\mathbb{R})$.