I think that the question is less about the notation and more about the philosophy of set theory, and in particular $ZFC$.
When coming to study set theory, you are not given a formal definition of the notion of a set. It might seem a bit circular, but a set is just an element of the universe of set theory.
You can have set theory with atoms, i.e. elements which are not sets, and you can have set theory without atoms, i.e. everything is a set.
What happens is that you understand from the real world, what is a set. It is a collection of things, many times it is a collection of things which you can describe nicely, i.e. there exists some sentence which is true for all the things in your collection, and only for them.
Examples can be "All the apples in this house." to describe a collection of apples, "All the people registered to math.SE whom are taller than 1.77m", to describe a certain collection of users on this website.
In mathematics however, it turns out that just taking arbitrary collections can be problematic, things like "The collection of all collections", and "The collection of all collections which are not in themselves", etc. give rise to some problems when defined formally into the language of set theory (the language which describes the $\in$-relations as "being an element of ...")
The result is that with respect to a certain model of the axioms of (some) set theory, a set is just a collection which happens to be an element of the universe.
For example, in $ZFC$ if there exists an inaccessible cardinal, then there is some $M\in V$, and $M$ is a transitive model of $ZFC$. Now take all the ordinals in $M$ - this is not a set in $M$, as the Burali-Forti paradox proves. However the ordinals in $M$ is a collection which is a set in $V$.
So to answer your question, in the $ZFC$ interpretation of set theory, a set is a collection of sets which is also a set. This is clear when reviewing the von Neumann construction of the universe:
$V_0 = \emptyset,\quad V_\alpha = \bigcup_{\beta<\alpha} \mathcal P(V_\beta)$
And the universe is simply "$V_{\mathbf{Ord}}$" where $\mathbf{Ord}$ is the class of all ordinals.
So what is an element in this universe? It appears at some stage $\alpha$ for the first time: If it was the empty set, then it is a set by definition. Otherwise it is a set of things which appeared before - and thus sets.
We have, if so, that every set is either empty, or all its elements are sets.