There are other ways of doing this integral, but let me try to fix your attempt, which is certainly a fine idea as far as it goes.
The main problem I spot with your development is that you forgot to change the $dx$ when you did the change of variable. (And you should be able to evaluate $\sec(\arctan a)$ as well; we'll get to that shortly).
So: you start with
$$\int_0^8 \sqrt{x^4+4x^2}\,dx = \int_0^8 \sqrt{x^2(x^2+4)}\,dx.$$
Then you do the change of variable $x=2\tan(\theta)$. If you do this, then
$$dx = 2\sec^2\theta\,d\theta;$$
when $x=0$, you want $\theta=0$, and when $x=8$ you want $\theta=\arctan(4)$ (you are correct there). So the integral actually becomes, after changing integrand, limits, and the $dx$:
$$\begin{align*}
\int_0^8\sqrt{x^2(x^2+4)}\,dx &= \int_0^{\arctan(4)}\sqrt{4\tan^2\theta(4\tan^2\theta+4)}2\sec^2\theta\,d\theta\\
&= \int_0^{\arctan(4)} \sqrt{16\tan^2\theta(\tan^2\theta+1)}2\sec^2\theta\,d\theta\\
&= \int_0^{\arctan(4)}8\sec^2\theta\sqrt{\tan^2\theta\sec^2\theta}\,d\theta\\
&= 8\int_0^{\arctan(4)}\sec^2\theta|\tan\theta\sec\theta|\,d\theta.
\end{align*}$$
Now, on $[0,\arctan(4)]$, both tangent and secant are positive, so we can drop the absolute value signs (something else you were not careful with), and the integral becomes
$$8\int_0^{\arctan(4)}\sec^3\theta\tan\theta\,d\theta.$$
Set $u=\sec\theta$. Then $du=\sec\theta\tan\theta$, so we have
$$\begin{align*}
8\int_0^{\arctan(4)}\sec^3\theta\tan\theta\,d\theta &= 8\int_{\sec(0)}^{\sec(\arctan(4))}u^2\,du\\
&= \frac{8}{3}u^3\Biggm|_{\sec(0)}^{\sec(\arctan(4))}\\
&=\frac{8}{3}\left(\sec^3(\arctan(4)) - \sec^3(0)\right).
\end{align*}$$
Now, $\sec(0) = 1$. What about $\sec(\arctan(4))$?
Say $\psi$ is an angle with $\tan(\psi)=4$. Take a right triangle with this angle; by scaling, we may assume the opposite side has length $4$ and the adjacent side has length $1$. Then the hypotenuse has length $\sqrt{17}$, so the cosine of $\psi$ is $\frac{1}{\sqrt{17}}$, hence the secant has value $\sqrt{17}$. So $\sec(\arctan(4)) = \sec(\psi) = \sqrt{17}$. Thus, the integral is:
$$\begin{align*}
\int_0^8\sqrt{x^4+4x^2}\,dx &= \frac{8}{3}\left(\sec^3(\arctan(4)) - \sec^3(0)\right)\\
&=\frac{8}{3}\left( \sqrt{17}^3 - 1^3\right)\\
&= \frac{8}{3}\left(17\sqrt{17} - 1\right).
\end{align*}$$
In summary: your mistake was that when you did the change of variable, you forgot to change the differential as well; and at the end you could have simplified $\sec(\arctan(4))$.
Of course, the better way of doing this is to factor out $x$ from the square root, and then recognize that you can do
$$\int_0^8\sqrt{x^4+4x^2}\,dx = \int_0^8x\sqrt{x^2+4}\,dx$$
with the change of variable $u=x^2+4$, like DJC suggested. But I thought you might like to know where exactly your approach went wrong (the substitution), and whether it could be brought to a correct conclusion (it could).