Your statement is somewhat imprecise. Your expression "can be factored as the product of two polynomials" should be replaced by "is reducible" (over $\mathbb Z$ or $\mathbb Q$), which excludes the possibility of the factors being invertible elements in their respective rings. Otherwise the statement would be trivially true, since any polynomial can be written as $1$ times itself.
The primitivity condition in the statement thus corrected is indeed superfluous. In fact, a polynomial is reducible over $\mathbb Z$ if and only if it is reducible over $\mathbb Q$ or it is not primitive. (This is the contrapositive of the statement of Gauss's lemma in the Wikipedia article others have linked to.) Thus, in a sense, the polynomial being primitive is not a condition for applying the lemma, but for needing the lemma, since for a non-primitive polynomial you know that it's reducible over $\mathbb Z$ without knowing whether it's reducible over $\mathbb Q$.