The group you are talking about is $[H, \phi].$ This group, though it does not really have a special name, is well known to group-theorists and goes back much further than the reference given by Arturo. More generally, if $A$ is any group of automorphisms of the group $G,$ then the group $[G,A]$ is the group generate by $\{ g^{-1}g^{a}: g \in G, a \in A \}.$ Here the result of the automorphism $a$ on $g$ is denoted by $g^{a},$ which is consistent with thinking of $[G,A]$ as a subgroup of the semidirect product $GA.$ Denotin, as is standard, $g^{-1}g^{a}$ by $[g,a]$ is is easy to check that $[g,a]^{h}[h,a] = [gh,a]$ for all $a \in A$ and $g,h \in G$ while also $[g,ab] = [g,b][g,a]^{b}$ for all $g \in G$ and $a,b \in A.$ This shows that for any $a \in A$, $[G,a]$ is a normal subgroup of $G$ ( so $[G,A] \lhd G$ also), and $[G,A]$ is $A$-invariant. This (sketchy) discussion is all at least implicit in D. Gorenstein's 1968 book "Finite Groups", for example. Since $H$ is not $\phi$-invariant in your question, we need to be a little more careful. Since $[xy,\phi] = [x,\phi]^{y}[y,\phi]$ for $x,y \in H,$ it does follow that $[H.\phi]$ is normalized by $H$, though it need not be a subgroup of $H.$ Similarly, the equation $[h, \phi^{2}] = [h,\phi][h,\phi]^{\phi}$ implies that $[H, \langle \phi \rangle ]$ is a $\phi$-invariant subgroup. For the moment, I am unsure whether $[H, \phi] = [H, \langle \phi \rangle]$ in general, though this is true when $H$ is $\phi$-invariant-(added later)-indeed, as Jack Schmidt points out, $[H,\phi] \neq [H, \langle \phi \rangle ]$ in general.