I would start by rotating the circle such that the given line becomes parallel to the $x$-axis. That way it just boils down to integrating the function $y=\sqrt{R^2-x^2}-c$ in some range for some constant $c$. In details:
How far is the line away from the origin? The line through the origin, perpendicular to the given line is parametrised by $y=\frac{b}{a}x$. It intersects the given line at $x=\frac{aR^2}{2(b^2+a^2)}$ and $y=\frac{bR^2}{2(a^2+b^2)}$. Hence the distance is $c=\frac{R^2}{2\sqrt{a^2+b^2}}$.
Edit: If $a=0$, the above computation makes obviously no sense, but then we don't have to perform it anyway since the line is parallel to the $x$-axis in the first place.
So the area is just the integral
$\int_{-x_0}^{x_0}\sqrt{R^2-x^2}-c\ dx$
To find the suitable range of integration, we have to find the zeros of $\sqrt{R^2-x^2}-c$. They are given by $x_0=\pm\sqrt{R^2-\frac{R^4}{4(a^2+b^2)}}.$
This is all a bit sketchy, but I am sure that you can figure out the details. If not feel free to ask.