There are several ways to think about the process. You are using the Substitution Rule, one version of which says \int g'(x)h'(g(x))\,dx = h(g(x))+ C. (\ast) We can easily verify that the result is correct. Just differentiate the expression on the right-hand side. By the Chain Rule, you get g'(x)h'(g(x)). So the Substitution Rule $(\ast)$ is just a rewritten version of the Chain Rule.
Now look at the expression you are trying to integrate. If you let $g(x)=8-x^3$, and h'(u)=u^5, then what you are trying to integrate looks very much like $(\ast)$. The only difference is that g'(x)=-3x^2, and in our integral we instead have $x^2$.
So we wish we had $-3x^2$ instead of $x^2$. Easily done, the two differ only by multiplication by a constant. So replace $x^2$ by $-3x^2$. You have multiplied your integral by $-3$. To make sure we have done nothing to the integral, we also divide by $-3$. Thus we arrive at the integral $\frac{1}{-3}\int (-3x^2)(8-x^3)^5\,dx.$
Now let $g(x)=8-x^3$, and h'(u)=u^5. Then $h(u)=\dfrac{u^6}{6}$. Apply $(\ast)$. We get $-\frac{1}{3} h(g(x))=-\frac{1}{3}\frac{(8-x^3)^6}{6}.$ Finally, don't forget to add the arbitrary constant of integration.
An equivalent version of $(\ast)$ which is more generally useful is \int g'(x)f(g(x))\,dx =\int f(u)\,du \qquad (\ast\ast) where $u=g(x)$. The evaluation goes much like before. Let $u=g(x)$, and $f(u)=u^5$. Then g'(x)=-3x^2, so we want -\frac{1}{3}\int g'(x)f(g(x)\,dx.
By $(\ast\ast)$ this is $-\frac{1}{3}\int f(u)du.$ But $\int f(u)\,du = \int u^5\,du,$ and now things are straightforward.
Shorthand version: Here is a very commonly used shorthand for doing essentially the same calculation. Let $u=8-x^3$. We want to express everything in terms of $u$, so that $x$ entirely disappears.
We have $\dfrac{du}{dx}=-3x^2$, "and therefore" $du =(-3x^2)dx$. It follows that $x^2\,dx=-\frac{1}{3}du$. "So" $\int x^2(8-x^3)^5\,dx=\int -\frac{1}{3}u^5 du=-\frac{1}{18}u^6+C.$ Finally, replace $u$ by $8-x^3$.
The above manipulation will not (and should not) make full intellectual sense to you. But it really is only a cleverly disguised version of $(\ast)$. After a while, if you get used to it, you will find that the above substitution process produces the right answer almost mechanically.
Important: Please remember that after you have found an integral, you can check easily that your answer is right, by differentiating. When you integrate, it is all too easy to be off by a minus sign, or more generally by a constant factor. Differentiating will usually reveal these kinds of slips.
A mnemonic: The $8-x^3$ is kind of ugly. So replace it by $g(x)$, or better by $u$ (which, of course, stands for "ugly"). We wish our integral had g'(x), that is, $\dfrac{du}{dx}$ sitting inside it.
Well, the derivative of $u$ is sitting in our integral. OK, not quite, the derivative of $u$ is $-3x^2$ and what we have in our integral is $x^2$. Easy fix, we multiply and divide by $-3$.