In the group $G$ with a binary operation $\cdot:G\times G\to G$ it is given by definition that $g\cdot h \in G$ for any $g,h\in G$. Since $\sqrt{2}\in S$ it should hold that if $(S,\cdot)$ is a group then $\sqrt{2}\cdot \sqrt{2}\in S$ but clearly $\sqrt{2}\cdot\sqrt{2}\in \mathbb Q$.
Here $\mathbb Q$ stands for rationals and $S$ for irrationals.
Some few words about the related things. The group $G$ is the set with a special structure: for any two elements $g,h$ of this set there exists a third element $g\cdot h$ which is also an element of $G$. There is also the unitary operation of taking inverses, i.e. for any $g\in G$ there exists $g^{-1}$ which also has to belong to $G$. Then you say that $G$ is closed under the group operation $\cdot$ and taking inverses.
When you define the group from the beginning this is quite clear, because you define such operation exactly with the set $G$ as a co-domain, i.e. $\cdot:G\times G\to G$ and $^{-1}:G\to G$. On the other hand, there may be a subset of $G$, say $H$ which is also closed under the group operation and taking inverses, so $h_1\cdot h_2\in H$ and $h^{-1}\in H$ for any $h,h_1,h_2\in H$. Such a subset of $G$ is called then a subgroup, so subgroup of $G$ is any subset of $G$ (together with the group operation and inverse defined as for $G$) which is closed under the group operation and taking inverses.
For example:
You define multiplication for all real numbers. Does it mean that $(\mathbb R,\cdot)$ is a group? On the one hand, multiplication of any two reals is real, but you cannot take inverse of $0$. So $(\mathbb R,\cdot)$ is not a group.
Now you may wonder if $(\mathbb R\setminus \{0\},\cdot)$ is a group. It is, because for any two non-zero reals their multiplication and inverse are defined and non-zero, so $\mathbb R\setminus \{0\}$ is closed under multiplication and taking inverses. You conclude that $(\mathbb R\setminus \{0\},\cdot)$ is a group (I omit here discussions about such properties as associativity).
What about the subgroups of $(\mathbb R\setminus \{0\},\cdot)$? First candidate is the set of negative reals $(\mathbb R_{<0},\cdot)$. Clearly, it is closed under taking inverses - but not under multiplication since $(-1)\cdot (-1) = 1\notin \mathbb R_{<0}$. Second candidate is the set of positive reals $(\mathbb R_{>0},\cdot)$ which is closed as under multiplication as under taking inverses.
Now, $(\mathbb R_{>0},\cdot)$ is a subgroup of $(\mathbb R\setminus \{0\},\cdot)$. Your question asks if $(S,\cdot)$ is also a subgroup - which we were able to disprove.
Finally, one example of the set which is closed under multiplication but not taking inverses is the set of natural numbers $\mathbb N$.