The basic idea of differential geometry is that you can differentiate functions in the spaces that you consider. "Smoothness" is simply the concept that there are functions that can be differentiated infinitely often.
The simplest functions are curves aka paths, i.e. functions from the reals (or an interval) to the model space $E$ you consider: $ \gamma: [a, b] \to E $ In order to define a differential quotent for such a curve you need to be able to write down $ \lim_{h \to 0} \frac{1}{h} (\gamma(t+h) - \gamma(t)) $ Define a path to be differentiable if all these limits exist.
In order for this to make sense, you need to have addition and scalar multiplication on $E$ and a topology so that the algebraic operations are continuous. This means that $E$ needs to be a topological vector space. Manifolds can then be defined to have locally homeomorphisms to your model space $E$.
If you use Hilbert spaces, you get Hilbert manifolds, the same holds true for Banach manifolds and Frechet manifolds. In finite dimensions, every topological vector space is algebraically isomorph to $\mathbb{R}^n$ and isomorph as topological vector space to $\mathbb{R}^n$ with its canonical topology, so there is not much choice.
Next, you can define smooth paths to be paths that are differentiable infinitely often, and define smooth maps of manifolds to be maps that map smooth paths to smooth paths.
This results in a definition of smooth/differential geometry in infinite dimensions.
For more details, see the book:
- Krigl, Michor: "The Convenient Setting of Global Analysis"
In this setting we characterize smooth maps of manifolds via the images of smooth paths.
It is possible to generalize this to more general functions than smooth paths and get, for example, diffeological spaces.