Let $S=\{p\in\Bbb Q:0
any non-empty open set. Suppose that $U$ is a non-empty open subset of $S$; the open intervals form a base for the topology of $\Bbb R$, so $U$ must contain a non-empty open interval $(a,b)$. But then $a$ and $b$ are real numbers such that $(a,b)\subseteq S\subseteq\Bbb Q$, which is impossible: there is certainly an irrational number between $a$ and $b$.
For essentially the same reason your set $A$ is not open. If $(a,b)$ were a non-empty open interval contained in $A$, every real number between $a$ and $b$ would be irrational, which is of course false.
And since neither of these complementary sets is open, neither can be closed: if $S$ were closed, $A$ would be open, and if $A$ were closed, $S$ would be open. Thus, both $S$ and $A$ fail as badly as possible to be clopen.
I know of no single term that means not clopen; one simply says that $S$ is not clopen, unless one can specify more precisely that $S$ is not open, that $S$ is not closed, or (as in this case) that $S$ is neither open nor closed.
Finally, note that as André already mentioned, a non-open set does not necessarily have an open complement. A set with an open complement is closed, and there are many sets that are neither open nor closed. A few people have studied the class of door spaces, spaces in which every set is either open or closed (or both), but this is a very restricted class (and not really a very interesting one). Most spaces of interest are not door spaces.