Well, first of all, if the derivative of order $k$-$1$ is not continuous, then the $k$-th derivative does not even exist.
As for the last one (the $k$-th one), you have a point: why do we need that derivative to be continuous. If we drop this requirement, wouldn't we get a broader class of functions, maybe able to represent a broader class of phenomena? In some sense yes. But...
While working with $C^k$ spaces, we are often interested in the pointwise value of some function $f$. If the 3rd derivative of $f$ has a physical/sociologic/demographic/whatever meaning, then it would be probably not satisfying if that derivative jumped at a point $x_0$. I said probably, since there are phenomena involving quantities that are allowed to change suddenly. In physics, for instance, the electric charge density can vary discontinuously. This quantity, in dimension one, turns out to be the derivative of the electric field, which itself is the derivative of the electric potential. Therefore, for this phenomenon, a potential which is $C^2$ according to the standard definition of $C^k$ spaces, might not represent the phenomenon that I am observing, since it would give a continuous charge density.
Nevertheless, functions that have $k$ continuous derivatives and the ($k$+$1$)-th has only jump discontinuity, form a pretty interesting space, called Hölder space with exponent $1$ (written $C^{k,1}$). Hölder spaces are pretty complicate, in my opinion, but the case $k=0$ is fairly easy: corresponds to Lipschitz-continuous functions, which are kind of useful in some contest. Maybe these spaces are closer to what you would expect.
But as I said, in many applications you are interested in the pointwise value of a function, and you don't expect that function to jump.
Things become more interesting when you don't care what's the value of the function at any given point, but rather you care that some other property holds, like for instance
$\int_a^b |f(x)|^2dx<\infty$
But this is completely another story...