Given vectors $\vec{a}=(3,-2,1),\vec{b}=(1,1,-4)$. Determine the angle between a plane that contains vectors $\vec{a},\vec{b}$ and a plane that contains vectors $\vec{a},\vec{b}+\vec{a}\times \vec{b}$.
There is the equation for angle between two planes when their normal vectors are given. How to evaluate it in this case? It is not specified if $\vec{a},\vec{b}$ are normal vectors or not.
A plane contains a vector if it lies in the plane, so for one thing such a vector is perpendicular to the normal to the plane. Therefore, since $\vec{a}$ and $\vec{b}$ are linearly independent, the plane containing $\vec{a}$ and $\vec{b}$ has normal $\vec{a} \times \vec{b}$.
The angle between planes with normals $\vec{n}_1$ and $\vec{n}_2$ is the same as the angle between the normals, i.e. $$\arccos{\frac{\vec{n}_1 \cdot \vec{n}_2}{\lVert \vec{n}_1 \rVert \lVert \vec{n}_2 \rVert}}. $$
With that said, in this case we can make the calculation a lot simpler by doing some algebra before putting the actual vectors in. The normal to the second plane is given by $\vec{a} \times \vec{b} + \vec{a} \times (\vec{a} \times \vec{b})$. The norm of this can be simplified since the two vectors in this sum are perpendicular, so $$ \lVert \vec{a} \times \vec{b} + \vec{a} \times (\vec{a} \times \vec{b}) \rVert^2 = \lVert \vec{a} \times \vec{b} \rVert^2 + \lVert \vec{a} \times (\vec{a} \times \vec{b}) \rVert^2 $$ This also means that the dot product simplifies: $$ (\vec{a} \times \vec{b}) \cdot (\vec{a} \times \vec{b} + \vec{a} \times (\vec{a} \times \vec{b})) = \lVert\vec{a} \times \vec{b} \rVert^2 + 0 $$
Therefore the cosine of the angle is given by $$ \frac{\lVert\vec{a} \times \vec{b} \rVert^2}{\lVert\vec{a} \times \vec{b} \rVert (\lVert \vec{a} \times \vec{b} \rVert^2 + \lVert \vec{a} \times (\vec{a} \times \vec{b}) \rVert^2)^{1/2}} = \frac{1}{(1+\lVert \vec{a} \times (\vec{a} \times \vec{b}) \rVert^2/\lVert \vec{a} \times \vec{b} \rVert^2)^{1/2}} $$
But for any vectors, $$ (\vec{x} \cdot \vec{y})^2 + \lVert \vec{x} \times \vec{y} \rVert^2 = \lVert \vec{x} \rVert^2\lVert \vec{y} \rVert^2, $$ so $$ \lVert \vec{a} \times (\vec{a} \times \vec{b}) \rVert^2 = \lVert \vec{a}\rVert^2 \lVert \vec{a} \times \vec{b} \rVert^2 - ( \vec{a} \cdot (\vec{a} \times \vec{b}) )^2 = \lVert \vec{a}\rVert^2 \lVert \vec{a} \times \vec{b} \rVert^2. $$ Therefore, we find the answer is $$ \arccos{\frac{1}{(1+\lVert \vec{a} \rVert^2)^{1/2}}} $$
I suggest that this is plausible: the change between the two planes is adding a multiple of $\vec{a} \times \vec{b}$, and everything in the angle expression scales in the same way with $\vec{b}$, but this term introduces an extra $\vec{a}$. Adding $c \vec{a} \times \vec{b}$ instead gives $1+c^2\lVert\vec{a}\rVert^2$ in the square root, which makes sense: if the two planes are the same, the angle is zero, and adding more will make the angle larger, and so the cosine smaller, but in this way we can't get more than a right angle.
The above may be rather unclear, so I suggest drawing a picture. Also, consider the case when $\vec{b}$ is chosen perpendicular to $\vec{a}$, so $\vec{a},\vec{b},\vec{a} \times \vec{b}$ is an orthogonal basis of $\mathbb{R}^3$, and draw a picture of the two planes. Then note that replacing $\vec{b}$ by $\vec{b}-k\vec{a}$ doesn't change the calculation in the slightest, so we may choose $\vec{b}\perp\vec{a}$ anyway.