I do not think "changing the operation to subtraction mod $4$" is actually well-defined. This is because $a-b\neq b-a$ unless $a=b$. So, what you need to do is something like, $h\ast g:=hg^{-1}$ which is not usually associative as $(h\ast g)\ast k=hg^{-1}k^{-1}$ while $h\ast (g\ast k)=h\ast (gk^{-1})=hkg^{-1}$. However, this structure has a right identity element, and every element has an inverse (which is itself: $g\ast g=gg^{-1}=1$). An alternative construction would be $h\ast g:=h^{-1}g^{-1}$ but this is actually an isomorphism (see if you can spot it! It is similar to something called the "opposite" group, which is the group under the operation $h\ast g:=gh$ and the opposite group is isomorphic to the group itself (the same is not true for other "opposites", such as opposite rings.))
Anyway, we shall think about the first operation: $h\ast g:=hg^{-1}$. mixedmath has pointed out in his answer that $G$ under this is either a quasigroup or a magma. It is, indeed, a quasigroup, as if $g\ast a=h$ then the only possibility for $a$ is $h^{-1}g$, while if $b\ast g=h$ then the only possibility for $b$ is $hg^{-1}$. (Look up the definition on, say, wikipedia if you are not sure what one is - basically, these $a$ and $b$ must exist and be unique.)
Note that for this operation to be associative you would need $kg^{-1}=g^{-1}k^{-1}$ for all $g, k\in G$ which means every element must have order $2$ (take $g=1$), and so $g^{-1}=g$ so the operation is just the operation of $G$! Note that if every element of $G$ has order $2$ then $G$ is necessarily abelian, so this is a boring case!
Again, this operation is commutative if and only if every element of $G$ has order $2$, as $h\ast g=hg^{-1}$ while $g\ast h=gh^{-1}$ so we need $hg^{-1}=gh^{-1}$ so $hg^{-1}hg^{-1}=1$. Taking $g=1$...