The volume of a sphere can be derived by integration using the disk method. In the same way we can derive the volume of the "sphere corner" you are looking for, but instead of using complete disks (circles), we use circular segments. In your figure, imagine slicing the sphere to the right of $x=a$ into thin slices. Each slice would be a circle and the part of the circle which was in the sphere corner would be a circular segment. We now intend to add these segments together by integrating from the start of the corner ($x=a$) to the end of the corner ($x=\sqrt{R^2-b^2}$).
In the figure below I've shown a circular segment:

From the previous link we find that the area of a circular segment can be expressed as: $$A_{seg}=R_1^{2}*\cos^{-1}(\frac{r_1}{R_1})-r_1* \sqrt{R_1^2-r_1^2}$$
We convert this to our notation by seeing that $r_1 = b$ and that $y=R_1$, giving: $$A_{seg}=y^{2}*\cos^{-1}(\frac{b}{y})-b* \sqrt{y^2-b^2}$$
As $y=\sqrt{R^2-x^2}$, we can now write the integral we need: $$V=\int_{a}^{\sqrt{R^2-b^2}} \, (R^2-x^2)*\cos^{-1}(\frac{b}{\sqrt{R^2-x^2}})-b* \sqrt{(R^2-x^2)-b^2} \, dx$$
Not a pretty integral. And when I enter it into Wolfram-Alpha, the computational time limit is exceeded. However, Wolfram-Alpha can provide the indefinite version of the integral, which is...uh...also not pretty. But, if you have the patience, you can insert the limits of the integral (i.e. $x=a$ and $x= \sqrt{R^2-b^2}$) into Wolfram's result and you have your answer!
EDIT
I've noticed a slight problem. When $x=\sqrt{R^2-b^2}$ we get divisions by zero in the Wolfram result. As a consequence I've decided to actually write the complete Wolfram result, though with simplifications which Wolfram didn't make. Here it is: $$\int \, (R^2-x^2)*\cos^{-1}(\frac{b}{\sqrt{R^2-x^2}})-b* \sqrt{(R^2-x^2)-b^2} \, dx = \\ -\frac{2}{3}bx \sqrt{-b^2+R^2-x^2}+ \frac{1}{3}b(3R^2-b^2)\tan^{-1}(\frac{-x}{\sqrt{-b^2+R^2-x^2}})+ \\ \frac{2}{3}R^{3}\tan^{-1}(\frac{bx}{R \sqrt{-b^2+R^2-x^2}})- \frac{1}{3}x(x^2-3R^2)\cos^{-1}(\frac{b}{\sqrt{R^2-x^2}})$$
That took half an hour. Anyway, if one looks at where the division with zero occurs, it is in the $tan^{-1}$ parts. If one analyses these parts, one finds for the first instance that $tan^{-1}=>-\frac{\pi}{2}$ as $x=>\sqrt{R^2-b^2}$, and for the second instance that $\tan^{-1}=>\frac{\pi}{2}$ as $x=>\sqrt{R^2-b^2}$. The equation above can therefore be used to get the volume V as a function of a and b, provided of course that $a^2+b^2 \lt R^2$.