Be careful that μ(H) can mean different things in the literature. I'll use the more specific μ(H,G) to indicate the Möbius function for the interval between H and G in the subgroup lattice of G. Philip Hall uses μ(H) for μ(H,G) in his paper:
Hall, P. "The Eulerian functions of a group." Quart. J. Math., Oxf. Ser. 7, 134-151 (1936). JFM:62.0082.02 DOI:10.1093/qmath/os-7.1.134
In order to calculate μ(H,G), I first calculate μ(1,G). This will allow me to use some nice reductions.
My copy of Hall is at the office, but I remembered the paper I was reading and it appears to be perfect for getting started with Möbius functions on groups (especially with regards to getting into the later rep theory):
Hawkes, T.; Isaacs, I. M.; Özaydin, M. "On the Möbius function of a finite group." Rocky Mountain J. Math. 19 (1989), no. 4, 1003–1034. MR1039540 DOI:10.1216/RMJ-1989-19-4-1003
This gives a remarkably useful reduction formula (Cor 3.3) for:
μ(1,G) = −k⋅μ(1,G/A)
where A is any abelian minimal normal subgroup of G with exactly k complements.
This formula (in its Eulerian form) is due to:
Gaschütz, Wolfgang. "Die Eulersche Funktion endlicher auflösbarer Gruppen." Illinois J. Math. 3 (1959) 469–476. MR107670 URL:euclid.ijm/1255455454
Using this reduction formula trivializes the cyclic and dihedral groups, along with the following observation on p. 1010 of Hawkes, et al. The Möbius function μ(1,H) does not depend on the embedding of H in G, it is an invariant of the isomorphism type of the group H. In particular, we can forget about the surrounding group G and just think about the cyclic group and the dihedral group (rather than cyclic subgroups or dihedral subgroups).
If G is a cyclic group of non-square-free order, then it has a cyclic Sylow p-subgroup P that is not order p. The only minimal normal subgroup of G contained in P is Ω(P), the subgroup of G consisting of the elements of order dividing p. This subgroup has no complements at all (just like 2Z/4Z is not complemented in Z/4Z, because Z/4Z has only a single involution). Hence k = 0 and μ(1,G) = −0⋅μ(1,G/Ω(P)) = 0. The normal Möbius function of an integer is 0 if the integer is not square-free.
If G is cyclic of square-free order, then each of its Sylow p-subgroups is order p, and so a minimal normal subgroup P. Each such subgroup P has exactly one complement (the Hall P′-subgroup of G consisting of those elements of order coprime to p). Hence k = 1 and we have μ(1,G) = −μ(1,G/P). Since G/P is a cyclic group of square-free order with one less prime factor we get the formula: μ(1,G) = (−1)j where |G|=n is a square-free number with exactly j prime factors. This is exactly the formula for μ(n).
Now if G is dihedral of order 2n, then μ(1,G) = −n⋅μ(n), again by considering a minimal normal subgroup of order p, which has p complements, unless p2 divides n.
Now we handle the case of μ(H,G). If N is a normal subgroup of G contained within H, then μ(H,G) = μ(H/N,G/N) by Noether's lattice isomorphism theorem.
This handles the cyclic groups immediately, since H=N is normal, we just get μ(H,G) = μ(1,G/H) = μ([G:H]).
This also handles the cyclic subgroups (of rotations) in a dihedral group: if G is dihedral and H ≤ [G,G] then H is cyclic and normal in G, with a dihedral quotient (possibly of order 2 or 4). Hence μ(H,G) = μ(1,G/H) = −kμ(k), where [G:H]=2k.
If H is a dihedral subgroup of a dihedral group G, then N=H∩[G,G] is cyclic, [H:N]=2 and N is normal in G (N is the group of rotations in H). μ(H,G) = μ(H/N,G/N), and so we are left with the case that G is dihedral and H is a reflection (order 2 subgroup not contained within the group of rotations). In this case, H is contained only in dihedral subgroups, one for each divisor of n. In other words, the subgroup lattice from H to G is isomorphic to the divisor lattice of [G:H]. Hence μ(H,G) = μ([G:H]).