As commented by Thomas and Ross, the values you're putting into the inverse cosine function are not in the interval $[-1,1]$, so you're going to get complex answers out of your formula.
This is one way that I would go about deriving the corrected formula:
Consider a cylindrical water tank (tipped on its side) with length $L$, radius $r$, filled with water to a height $h$, as pictured in the link you provided.
To find the volume of the water, we will want to integrate the area of horizontal cross sections as we go along a vertical axis.
For simplicity, let's set up a vertical $y$-axis with 0 located at the center of the circle at the end of the tank.
Then the horizontal cross sections of the tank look like rectangles with length $L$, and a width which we can call $w$.
You can (kind of) see what this looks like in this poorly drawn diagram:
Now, using the Pythagorean Theorem, we find that at a certain height $y$ we have width $w = 2\sqrt{r^2-y^2}$, and so the cross-sectional area of the tank at height $y$ is given by $A = 2L\sqrt{r^2-y^2}$.
Now if we integrate this with respect to $y$ on the interval $[-r,h]$, we should get the volume up to height $h$.
We get the integral
$$
V = \int_{-r}^h 2L\sqrt{r^2-y^2} dy .
$$
We can evaluate this by using the trig substitution $y = r\sin \theta$.
Making this substitution, get
\begin{align*}
V
&= \int_{-\pi/2}^{\sin^{-1}(h/r)} 2L r\cos \theta \cdot r \cos \theta d\theta \\
&= 2L r^2 \int_{-\pi/2}^{\sin^{-1}(h/r)} \cos^2 \theta d\theta \\
&= 2L r^2 \int_{-\pi/2}^{\sin^{-1}(h/r)} \left( \frac 12 + \frac 12 \cos(2 \theta) \right) d\theta \\
&= 2L r^2 \left( \frac \theta 2 + \frac 14 \sin 2\theta \right) \bigg|_{-\pi/2}^{\sin^{-1}(h/r)} \\
&= L r^2 (\theta + \sin \theta \cos \theta ) \bigg|_{-\pi/2}^{\sin^{-1}(h/r)} \\
&= L r^2 \left( \sin^{-1} \left( \frac hr \right) + \frac \pi 2 - \frac hr \sqrt{r^2-h^2} \right) \\
&= L r^2 \sin^{-1} \left( \frac hr \right) + \frac \pi 2 L r^2 - Lrh \sqrt{r^2-h^2} .
\end{align*}