Although it is certainly important to study frequently seen group operations like permutations, function composition, word operations, and so on, I find it fascinating to see group structure applied in strange and complicated ways to other disciplines.
Lately MSE has got me looking at the group of functions $f:\mathbb{N}\rightarrow \mathbb{C}$ with $f(1)\not= 0$ under the Dirichlet product $\star$ given by $(f\star g)(n)=\sum_{ab=n}f(a)g(b).$ I think this is a great example of an unusual group operation. I don't usually get into this type of arcane number theoretic stuff and never would have thought to look at this type of structure myself.
What are some other examples of obscure group operations I may not have heard of?