The problem here is that symbol manipulation is being substituted for reasoning. The limit does not mean that we can simply instantiate the $x$ variable as the $\infty$ symbol and do some arithmetic. The reason that $\infty - \infty$ is indeterminate is that $\infty$ is not a specific number, but a concept. You've substituted a not-a-number for a variable and then concluded that the formula is indeterminate.
One way to rescue this is this: since you have substituted some concrete entity for $x$, both instances of that entity have to be the same manifestation of that entity. That is to say, your $\infty - \infty$ is not just any two infinities; it is the same infinity occurring twice: that infinity which was substituted for $x$. This infinity has to be equal to itself.
But of course there can be two different infinities in a formula obtained differently. For instance suppose we have $x - y$, and both $x$ and $y$ tend to infinity. Since they are independent, they go to different infinities, and so if we substitute, we get an indeterminate $\infty - \infty$. Here, each infinity is from a different substitution and so a different infinity object.
Our mathematical typography does not visually distinguish two infinities that are multiple occurences of the same infinity, and two infinities of independent origin. Both just look like $\infty$.
Ultimately, the limit concept is about approaching but not reaching. To ask whether there is a limit as $x\to\infty$ really means "does the formula converge on a value as $x$ gets arbitrarily large". There is no question of substituting some concrete infinity for $x$; it is a question of probing the space of $x$ to large values to see whether the difference between one formula and another (its supposed limit) can be shown to keep diminishing as $x$ moves toward the limit.
It is because limit means "approach, but do not reach", we can evaluate this:
$\lim_{x\to 0} {x\over x}$
The function $x\over x$ is exactly like $1$ except that it is not continuous over $x = 0$. It has no value there. Yet, the limit of this function as $x$ approaches zero (from either side) is 1.
We cannot calculate this limit by substituting $0$ for $x$, similarly to the difficulty of substituting $\infty$ for $x$ in a $lim_{x\to \infty}$ situation.
We can sometimes short-cut to a limit by substituting zero: in cases where we do not get an absurdity like division by zero.