A $d$-dimensional topological manifold $M$ is a topological space for which there exists continuous maps (charts) to $R^d$, such that every point $p \in M$ is mapped by at least one of those charts.
A differentiable manifold $M$ is a topological manifold with the additional requirement that its Atlas contains only charts that are diffeomorphic to eachother (i.e. whose transition maps are all differentiable).
I find this second definition (of differentiable manifolds) very strange, because it only makes reference to the charts in the Atlas of manifold $M$, but not to the manifold itself. I find it very difficult to imagine how this condition puts a constraint on $M$, because we can always remove charts from the Atlas, such that only charts that are diffeomorphic remain. This seems to be a property of the charts, but not of the underlying Manifold $M$.
Or do there exist manifolds for which it is impossible to remove non-diffeomorphic charts from the Atlas, and still have a set of charts that incorporates all points on the manifold? If so, I find this difficult to imagine.
So my question: How does the condition of the diffeomorphism of a manifold's charts put any constraints on that manifold?