The book is correct; your professor is correct when $d$ divides (i.e. goes into) $n$.
Note that for any $g\in G$,
$$|\langle g\rangle|=\text{ord}(g).$$
The general relation is that, if $\text{ord}(a)=n$, then
$$\text{ord}(a^d)=\frac{n}{\gcd(d,n)}.$$
For example, consider the cyclic group $G=\mathbb{Z}/6\mathbb{Z}=\{0,1,2,3,4,5\}$ with operation $+\,$, and let $a=1$. It has order $n=6$. Let $d=5$; then $a^5$ means
$$a+a+a+a+a=5$$
so $H=\langle a^5\rangle=\langle 5\rangle=G$, so $|H|=6=\frac{6}{\gcd(5,6)}$. In contrast, the statement that $|H|=\frac{6}{5}$ doesn't even make any sense.
If you intended $n$ to be the order of $G$, then both answers will be false unless $G$ is a cyclic group and $\langle a\rangle=G$ (in my answer above, I assumed you meant $n=\text{ord}(a)$, which agrees in this case, i.e. $\text{ord}(a)=n=|G|$ if and only if $G$ is cyclic and $G=\langle a\rangle$). For example, in the group $$G=(\mathbb{Z}/6\mathbb{Z})\times(\mathbb{Z}/2\mathbb{Z}),$$
for which $|G|=12$, the element $a=(1,1)$ has order 6, and
$$a^5=(1,1)+(1,1)+(1,1)+(1,1)+(1,1)=(5,1),$$
hence
$$H=\langle a^5\rangle=\langle(5,1)\rangle$$
has $|H|=6$, but
$$|H|\neq\frac{12}{1}=\frac{12}{\gcd(5,12)}$$
and
$$|H|\neq\frac{12}{5}$$