Like I said in my comment, I do not see any reason for this result to hold, so I tried to find a counter-example.
I didn't find an explicit one, but I did find what would not work, so I am sharing it here to prevent someone else to lose time in this direction.
I tried to find two groups where they would not be any elements of order $>3$. Then we would have two non isomorphic groups, with not the same number of elements of order $2$ and $3$ respectively and we would have won.
But such a groupe is of one of the following types (which both works as acceptable groups regarding to our conditions):
- Type $T$. The group is isomorphic to the set
$$(\mathbb Z/3\mathbb Z)^\Gamma\times \{\pm 1\}$$
with the product:
$$(h,a)\cdot (k,b)=(hk^a,ab)$$
where $(\mathbb Z/3\mathbb Z)^\Gamma$ is the set of maps from a given set $\Gamma$ (of cardinality $n$) to $\mathbb Z/3\mathbb Z$.
*We can note that if $n=1$, then a group of type $T$ is isomorphic to $\mathfrak S_3$.
- Type $S$. The group is isomorphic to the set
$$\left((\mathbb Z/2\mathbb Z)^2\right)^\Gamma\times \mathbb Z/3\mathbb Z$$
with the product:
$$(h,a)\cdot (k,b)=(h\cdot\alpha^a(k),a+b)$$
where we think of $\mathbb Z/3\mathbb 2$ as $\{0,1,2\}$ under addition modulo $3$, and $\alpha$ is a cyclic permutation of three nonidentity elements of $(\mathbb Z/2\mathbb Z)^2$.
*We can note that if $n=1$, then a group of type $S$ is isomorphic to $\mathfrak A_4$.
We can notice that a group of type $S$ and a group of type $T$ won't have the same order unless maybe if $n=1$, which won't work either.
Conclusion: there is no counter-example with of group with only elements of order $2$ and $3$.
I wish someone find a better result.
Sources : one article, another article and a final article.