Symmetries of a colored square

Question

Consider a square, colored black and white, with the following transformations allowed:

• rotate the square by 90 degrees,
• reflect the square about an axis, and
• swap black and white.

I seek to characterize the various ways a given square might have symmetry, and in particular the various ways that a square painted in certain ways might be transformed to not look identical.

For instance, there are a full complement of 16 ways that a white square with a black rectangle in the corner might be transformed:

Meanwhile, an all white square can only be changed to all black: no rotations or reflections work; and a square with the left half painted black has four variations, because two rotations and a color swap, or two rotations and a reflection, both give no transformation at all.

This system of transformations is the group $D_8\times Z_2$, with group presentation $G := \langle a,x,y \mid a^4 = x^2 = y^2 = e, xax = a^{-1}, ya = ay, xy = yx \rangle$. In this, $a$ is rotation, $x$ is reflection (which i'll say is reflection about the horizontal axis), and $y$ is color swapping.

I found that many such layout types are quotient groups: for instance, the square with a rectangle in it is the quotient $G / \langle 1 \rangle$, the all white one is $G / \langle a, x \rangle$, and the one with the left half black is $G / \langle a^2 y, a^2 x \rangle$ Thus, any normal subgroup that doesn't include $y$ as one of its generators (because it's impossible for $y$ alone to leave the square unchanged) should provide a suitable set of transformations.

But there are some layouts where, it seems, there is no quotient group to be had! Here's one of them: the square has only the left third painted black.

This has the following pairs of transformations giving the same thing: $e=x$, $a=a^3x$, $a^2=a^2x$, $a^3=ax$, $y=xy$, $ay=a^3xy$, $a^2y=a^2xy$, $a^3y=axy$. There's obviously only one subgroup of order $2$ whose quotient would include the first pairing, but that's 1. not a normal subgroup so doesn't provide a quotient and 2. would suggest $a=ax$ anyway, which isn't here.

Which finally gets me to my question: What symmetries are available here that are nonetheless not quotient groups, and how do I go about finding them?

Answer 1

Considering the last example, you have your group $G$ acting transitively on the given set $\Omega$ of $8$ elements.

The orbit-stabilizer theorem (see also this) tells you that the action of $G$ on $\Omega$ is similar (isomorphic as a $G$-set) to the action of $G$ on the (right, say, if you are using right actions) cosets in $G$ of the stabilizer $G_{\alpha}$ of any element of $\Omega$.

For instance if $\alpha$ is the first element, then $G_{\alpha} = \{e, x \}$.

So you won't find a quotient group in general, because $G_{\alpha}$ might not be normal in $G$, but a coset space instead.