Say that you have an orthonormal rotation matrix
$$ R = \left[ \begin{array}{ccc} R_{xx} & R_{xy} & R_{xz} \\ R_{yx} & R_{yy} & R_{yz} \\ R_{zx} & R_{zy} & R_{zz} \end{array} \right] $$
This matrix rotates a $3 \times N$ matrix of 3-dimensional column vectors $A$ in some arbitrary way about the X, Y, and Z axis as $A' = RA$. Starting from this point, how can I remove a rotation about an axis?
For example, this arises if I want to preserve a certain direction. Let's say my column vectors in the $A$ matrix amount to points in 3D space. They are all oriented on some plane with a normal vector $n$. I want to modify $R$ in such a way that the my modified rotation matrix $R'$ applies the transformation to my set of points $A$ without affecting my normal $n$. Put more simply, how can I modify $R$ to ensure that $A'$ and $A$ have the same normal vector $n$?
For the sake of argument, we can assume that $n = [0, 1, 0]$.