I was trying to think of an example where $gHg^{-1} \ne g^{-1}Hg$. I couldn't think of one, but I am curious if the following reasoning demonstrates that, at the very least, such an example must exist:
Let $H$ be a non-normal subgroup of $G$ and let $g \in G - H$. Then it is possible that $gHg^{-1} \ne g^{-1}Hg$ since if we supposed otherwise we would have
$\begin{align} gHg^{-1} & = g^{-1}Hg \\ g^2Hg^{-1} & = Hg\\ g^2 & = e\\ g & = e \end{align}$
which is impossible since we assumed $g \in G - H$.