There are other ways of doing this integral, but let me try to fix your attempt, which is certainly a fine idea as far as it goes.
The main problem I spot with your development is that you forgot to change the $dx$ when you did the change of variable. (And you should be able to evaluate $\sec(\arctan a)$ as well; we'll get to that shortly).
So: you start with $\int_0^8 \sqrt{x^4+4x^2}\,dx = \int_0^8 \sqrt{x^2(x^2+4)}\,dx.$
Then you do the change of variable $x=2\tan(\theta)$. If you do this, then $dx = 2\sec^2\theta\,d\theta;$ when $x=0$, you want $\theta=0$, and when $x=8$ you want $\theta=\arctan(4)$ (you are correct there). So the integral actually becomes, after changing integrand, limits, and the $dx$: $\begin{align*} \int_0^8\sqrt{x^2(x^2+4)}\,dx &= \int_0^{\arctan(4)}\sqrt{4\tan^2\theta(4\tan^2\theta+4)}2\sec^2\theta\,d\theta\\ &= \int_0^{\arctan(4)} \sqrt{16\tan^2\theta(\tan^2\theta+1)}2\sec^2\theta\,d\theta\\ &= \int_0^{\arctan(4)}8\sec^2\theta\sqrt{\tan^2\theta\sec^2\theta}\,d\theta\\ &= 8\int_0^{\arctan(4)}\sec^2\theta|\tan\theta\sec\theta|\,d\theta. \end{align*}$ Now, on $[0,\arctan(4)]$, both tangent and secant are positive, so we can drop the absolute value signs (something else you were not careful with), and the integral becomes $8\int_0^{\arctan(4)}\sec^3\theta\tan\theta\,d\theta.$ Set $u=\sec\theta$. Then $du=\sec\theta\tan\theta$, so we have $\begin{align*} 8\int_0^{\arctan(4)}\sec^3\theta\tan\theta\,d\theta &= 8\int_{\sec(0)}^{\sec(\arctan(4))}u^2\,du\\ &= \frac{8}{3}u^3\Biggm|_{\sec(0)}^{\sec(\arctan(4))}\\ &=\frac{8}{3}\left(\sec^3(\arctan(4)) - \sec^3(0)\right). \end{align*}$
Now, $\sec(0) = 1$. What about $\sec(\arctan(4))$?
Say $\psi$ is an angle with $\tan(\psi)=4$. Take a right triangle with this angle; by scaling, we may assume the opposite side has length $4$ and the adjacent side has length $1$. Then the hypotenuse has length $\sqrt{17}$, so the cosine of $\psi$ is $\frac{1}{\sqrt{17}}$, hence the secant has value $\sqrt{17}$. So $\sec(\arctan(4)) = \sec(\psi) = \sqrt{17}$. Thus, the integral is: $\begin{align*} \int_0^8\sqrt{x^4+4x^2}\,dx &= \frac{8}{3}\left(\sec^3(\arctan(4)) - \sec^3(0)\right)\\ &=\frac{8}{3}\left( \sqrt{17}^3 - 1^3\right)\\ &= \frac{8}{3}\left(17\sqrt{17} - 1\right). \end{align*}$
In summary: your mistake was that when you did the change of variable, you forgot to change the differential as well; and at the end you could have simplified $\sec(\arctan(4))$.
Of course, the better way of doing this is to factor out $x$ from the square root, and then recognize that you can do $\int_0^8\sqrt{x^4+4x^2}\,dx = \int_0^8x\sqrt{x^2+4}\,dx$ with the change of variable $u=x^2+4$, like DJC suggested. But I thought you might like to know where exactly your approach went wrong (the substitution), and whether it could be brought to a correct conclusion (it could).