It's mostly fine. However:
First. When you do a change of variable/substitution in a definite integral, you need to change the interval of integration.
Here you are setting $u=x^2+a^2$. That means that when $x=0$, you have $u=0^2+a^2 = a^2$; and when $x=a$, you get $u=a^2+a^2=2a^2$. So the integral will not be an integral from $0$ to $a$ when it is an integral with respect to $u$, it will be an integral from $a^2$ to $2a^2$. So you go from $\int_0^a x\sqrt{x^2+a^2}\,dx$ to $\frac{1}{2}\int_{a^2}^{2a^2}u^{1/2}\,du.$
Second. You just plug in: $\begin{align*} \int_0^a x\sqrt{x^2+a^2}\,dx &= \frac{1}{2}\int_{a^2}^{2a^2}u^{1/2}\,du &\text{(through all the work you've done)}\\ &= \frac{1}{2}\left.\left(\frac{2}{3}u^{3/2}\right|_{a^2}^{2a^2}\right)\\ &= \frac{1}{2}\left(\frac{2}{3}(2a^2)^{3/2} - \frac{2}{3}(a^2)^{3/2}\right)\\ &= \frac{1}{2}\frac{2}{3}2^{3/2}a^{2(3/2)} - \frac{1}{2}\frac{2}{3}a^{2(3/2)}\\ &= \frac{2^{3/2}}{3}a^3 - \frac{1}{3}a^{3}\\ &= \left(\frac{2^{3/2}}{3} - \frac{1}{3}\right)a^3\\ &= \left(\frac{2\sqrt{2}-1}{3}\right)a^3. \end{align*}$ That's it: since the problem is in terms of $a$, the answer will also be in terms of $a$. (I used the fact that $2^{3/2} = (2^{1/2})^3 = (\sqrt{2})^3 = (\sqrt{2})^2\sqrt{2} = 2\sqrt{2}$).
The alternative way is to do the indefinite integral first, and then use the antiderivative: to compute $\int_0^ax\sqrt{x^2+a^2}\,dx$ we first do the indefinite integral (using your substitution): $\begin{align*} \int x\sqrt{x^2+a^2}\,dx &= \frac{1}{2}\int u^{1/2}\,du &\text{(setting }u=x^2+a^2\text{)}\\ &= \frac{1}{2}\left(\frac{2}{3}u^{3/2}\right) + C\\ &= \frac{1}{3}u^{3/2}+C\\ &= \frac{1}{3}(x^2+a^2)^{3/2} + C. &\text{(going back to }x\text{)} \end{align*}$ Having done that, we can use this antiderivative (with any particular value for $C$) to obtain the original definite integral, via the Fundamental Theorem of Calculus: $\begin{align*} \int_0^a x\sqrt{x^2+a^2}\,dx &= \left.\left(\frac{1}{3}(x^2+a^2)^{3/2}\right|_0^a\right)\\ &= \left(\frac{1}{3}(a^2+a^2)^{3/2}\right) - \left(\frac{1}{3}(0^2+a^2)^{3/2}\right)\\ &= \frac{1}{3}(2a^2)^{3/2} - \frac{1}{3}(a^2)^{3/2}\\ &= \frac{2^{3/2}}{3}a^3 - \frac{1}{3}a^3\\ &= \frac{2\sqrt{2}-1}{3}a^3, \end{align*}$ same answer as above.
What you should not do is do a substitution in a definite integral and not correct the limits of integration. The way I tell my students this is to remember the following maxim:
If you change the variable, you have to change the limits.
You should not rely on leaving the limits as they were thinking "I'll just go back to $x$ when I'm done with the integral." That's just asking for trouble (like, forgetting to go back to $x$), and it may be difficult to unwind if you have to make several substitutions along the way (or use other methods of integration). Better to do the simple computations needed to change the interval (which amount to just plugging the limits into the substitution formula) and then not have to worry about it.