It really depends on the context.
For a lot of mathematics, I would agree with Brian M. Scott: in stuff like elementary real/complex analysis (and similar some basic constructions in algebra), one almost always work only with objects and sets of objects. In these situations as Dan Petersen comments, there is already some "established notation" where the line between $\{x\}$ and $x$ are blurred, and abusing notation this way may be "marginally acceptable".
For actually doing foundations stuff or set theory, then I agree with Mark Dominus. In a universe where everything is a set (or a proper class), and dealing with sets of sets and specifically power sets is common, you should absolutely not abuse notation like that. This is because if you define the set $y := \{ \{\}, x, \{x\}\}$, the two sets $y\setminus x$ and $y\setminus \{x\}$ are very different objects.
In other words, before you abuse the notation, you have to justify why the notation can be abused. If you work in a context where sets are always collections of some primitive objects, where sets of sets are not considered, then you may be able to get away with this notation by virtue of being able to identify $x$ with $\{x\}$. By in general I would advise against doing something like that.