To find the inverse of a function like this, you can switch the roles of $x$ and $y$ and solve the new equation for $y$. So, you would want to solve $x = -y^2 + 8y$ for $y$.
Rewriting that a little, we get $y^2 - 8y + x = 0$, which is a quadratic in $y$. That is, you would be taking $a = 1$, $b = -8$, and $c = x$ in the quadratic formula. Don't let the fact that $x$ is a variable bother you: the function $y^2 - 8y + x$ is quadratic in $y$ and that's all that matters here.
Applying the formula, you would get
$\begin{align*} y &= \frac{8 \pm \sqrt{64 - 4x}}{2}\\ &= \frac{8 \pm \sqrt{4(16 - x)}}{2}\\ &= \frac{8 \pm 2\sqrt{16 - x}}{2}\\ &= 4 \pm \sqrt{16 - x} \end{align*}$
The fact that you get $\pm$ in your solution means that the original function doesn't have an inverse on the entire real line. Depending on how you restrict the domain of the original function, your inverse can be either $4 - \sqrt{16 - x}$ or $4 + \sqrt{16 - x}$.