Note that, regardless of whether $AB$ is a subgroup or not, if both $A$ and $B$ are finite, then the number of elements it has is $|AB| = \frac{|A||B|}{|A\cap B|}.$
To see this, consider the map $A\times B \to AB$ given by $(a,b)\mapsto ab$. The map is clearly onto. If $x\in A\cap B$, then for each $a\in A$ and $b\in B$ you have $ax\in A$, $x^{-1}b\in B$, so $(ax,x^{-1}b)\in A\times B$ has the same image as $(a,b)$. Thus, each element of $AB$ is the image of at least $|A\cap B|$-many elements. Conversely, if $(a,b)$ and (a',b') have the same image under this map, then ab=a'b', hence (a')^{-1}a=b'b^{-1}\in A\cap B, and a = a'(a'^{-1}a) and b=(a'^{-1}a)^{-1}b' = (bb'^{-1})b', so any two pairs that map to the same element arise from an element of $A\cap B$. Thus, each image occurs $|A\cap B|$ times, so $|AB||A\cap B| = |A\times B|=|A||B|$.
In particular, take $G=S_3$, $A=\langle(1,2)\rangle$, and $B=\langle (1,3)\rangle$. Then $A\cap B = \{e\}$, so the number of elements in $AB$ is $|A||B|=4$. This cannot be a subgroup of $S_3$, by Lagrange's Theorem. So $AB$ is properly contained in $\langle A,B\rangle$ (as sets).