HINT with another approach, for the inner integral, substitute $\text{y}=x\tan\left(\text{u}\right)$:
$$\mathcal{I}\left(x\right)=\int_0^{\sqrt{2x-x^2}}\sqrt{x^2+\text{y}^2}\space\text{d}\text{y}=x^2\int_0^{\arctan\left(\frac{\sqrt{2-x}}{\sqrt{x}}\right)}\sec^3\left(\text{u}\right)\space\text{d}\text{u}\tag1$$
Now, use the reduction formula:
$$\int\sec^\text{n}\left(\text{u}\right)\space\text{d}\text{u}=\frac{\sec^{\text{n}-2}\left(\text{u}\right)\tan\left(\text{u}\right)}{\text{n}-1}+\frac{\text{n}-2}{\text{n}-1}\int\sec^{\text{n}-1}\left(\text{u}\right)\space\text{d}\text{u}\tag2$$
So, we get:
$$\mathcal{I}\left(x\right)=\frac{x^2}{2}\cdot\left(\int_0^{\arctan\left(\frac{\sqrt{2-x}}{\sqrt{x}}\right)}\sec\left(\text{u}\right)\space\text{d}\text{u}+\sec\left(\text{u}\right)\tan\left(\text{u}\right)\Big{|}_0^{\arctan\left(\frac{\sqrt{2-x}}{\sqrt{x}}\right)}\right)\tag3$$
Now, use that:
$$\int\sec\left(\text{u}\right)\space\text{d}\text{u}=\ln\left|\tan\left(\text{u}\right)+\sec\left(\text{u}\right)\right|+\text{C}\tag4$$
So, we get that:
$$\mathcal{I}\left(x\right)=\frac{x^2}{2}\cdot\left(\ln\left|\frac{\sqrt{2}+\sqrt{2-x}}{\sqrt{x}}\right|+\frac{\sqrt{\frac{4}{x}-2}}{\sqrt{x}}\right)\tag5$$