There is an invariant subspace that you have not considered : the one consisting of vectors with all their coordinates equal, i.e. the one generated by $(1,1,1)$. Clearly all the vectors in this subspace are fixed by the action, so the subspace is invariant. Moreover, the action preserves the scalar product on $\Bbb R^3$; hence the orthogonal subspace is invariant as well. This orthogonal subspace consists of all vectors such that $x+y+z=0$, so you can see directly that it has to be invariant.
By the way your intuition is right : the action correspond indeed to rotations by an angle of $\frac{2\pi}{3}$, but the axis of the rotation is the line generated by $(1,1,1)$. Indeed, you can take a vector in the orthogonal subspace $(a,b,-a-b)$, and check that it makes an angle $\alpha=\frac{2\pi}{3}$ with its image under the action, $(-a-b,a,b)$:
\begin{align}\cos \alpha & = \frac{(a,b,-a-b)\cdot (-a-b,a,b)}{\|(a,b,-a-b)\|\cdot\|(-a-b,a,b)\|}\\ & =\frac{-a^2-ab+ba-ba-b^2}{a^2+b^2+(a+b)^2}\\ & =\frac{-a^2-ab-b^2}{2a^2+2ab+2b^2}=\frac{-1}{2}\\ &=\cos \left(\frac{2\pi}{3}\right) .\end{align}
This gives us two non-trivial invariant subspaces. We can show that there cannot be other invariant subspaces. Indeed, if $U$ is an invariant subspace, then :
- either $U$ has dimension $1$ ; then $U$ is generated by a vector $(a,b,c)\neq 0$. Then $(c,a,b)\in U$ by invariance, and thus there exists $\lambda \in \Bbb R$ such that $(c,a,b)=\lambda (a,b,c)$. Then we have $a=\lambda b=\lambda^2 c=\lambda^3 a$, and the same holds for $b$ and $c$. Hence $\lambda^3=1$, and thus $\lambda =1$ and $a=b=c$, so $U$ is the subspace we already knew.
- either $U$ has dimension $2$. Then its intersection with the plane $x+y+z=0$ is also invariant, and has dimension at least $1$. If the intersection has dimension $1$, the previous point shows that it must be generated by $(1,1,1)$, which is impossible as it must also lie in the plane $x+y+z=0$. Thus the intersection has dimension $2$, and thus $U$ is the plane of equation $x+y+z=0$.
