I'm not sure exactly what you're asking, but the pseudoinverse can be well-understood in terms of the singular value decomposition. The SVD $A=U \Sigma V^*$ looks like, $A=U\left[\begin{matrix}\sigma_1 & & & & \\ & \sigma_2 & & & \\ & & \ddots & & \\ & & & \sigma_k \\ & & & & 0 \\ & & & & & \ddots \\ & & & & & & 0\end{matrix}\right]V^*$ where $U$ and $V$ are orthogonal matrices (eg, rotations and flips) and $\sigma_i$ are positive. In this form, the pseudoinverse $A^+=V \Sigma^+ U^*$ can be written as, $A^+ = V \left[\begin{matrix} 1/\sigma_1 & & & & \\ & 1/\sigma_2 & & & \\ & & \ddots & & \\ & & & 1/\sigma_k \\ & & & & 0 \\ & & & & & \ddots \\ & & & & & & 0\end{matrix}\right]U^*.$ This makes it pretty straightforward to characterize exactly in what way the pseudoinverse fails to be an inverse. We have,
$A^+A-I = U \Sigma V^* V \Sigma^+ U^* - UU^* = U (\Sigma \Sigma^+-I)U^* \\ = U \left[\begin{matrix}0 \\ & \ddots \\ & & 0 \\ & & & 1 \\ & & & & \ddots \\ & & & & & 1\end{matrix}\right]U^*.$
More to the point, consider a version of $A$ that doesn't map the whole space, but instead map from the space spanned by the first $k$ columns of $V$ (right singular vectors) to the space spanned by the first $k$ columns of $U$ (left singular vectors). On this restricted space, $A^+$ is an exact inverse, and on the complementary space $A^+$ is zero so it fails to be an inverse identically.