I know this is late - but I'd like to answer it anyways, for those who will run across this.
First, your initial integral looks a bit wrong. It looks like you've used angles as the limits as the limits of integration for the outer integral.
What I assume the problem wants (if you're wanting $y$ between those points) is something like:
$ \displaystyle \iint_R 3(x-y)\,\mathrm{d}A,$ where $R$ is the region in the first quadrant bounded (enclosed) by the semicircle $x = \sqrt{2 - y^2}$ and the lines $x = y$ and $x = 0$ (the y-axis).
Always start these kinds of problems by drawing a picture. If you aren't sketching the graph, you're not being careful enough.
So let's take the curves it gives us and graph them. First, it may be easier to rewrite $x = \sqrt{2 - y^2}$ as $x^2 + y^2 = 2$, which is much easier to look at when we're trying to graph - (it's just a circle centered at the origin with radius $\sqrt2$).
Sorry for the primitive drawing - I don't quite know mathematica well enough yet to get a good plot.

So the shaded area is what you're looking for. To describe this region in polar coordinates, we need two things specifically: the range of the angle and the radius over which we'd like to integrate.
Looking at the equation for the circle we can see that the radius of the curve is $\sqrt2$. There's also nothing bounding the lower end of the radius, so we'll use the full radius of the circle: $0 \le r \le \sqrt2$.
To find the angle, look at the shaded part of the graph. You can see that $x=y$ makes an angle of $\displaystyle \frac{\pi}{4}$ radians with the x-axis, and $x = 0$, being the y-axis, is at an angle of $\displaystyle \frac{\pi}{2}$ radians.
So we can describe the region over which we wish to integrate as:
$R = \{(r,\theta)\,|\,\frac{\pi}{4}\le\theta\le\frac{\pi}{2}, 0 \le r \le \sqrt2\}$
And finally, rewriting the function in polar form we get the double integral
$\displaystyle \int_{\pi/4}^{\pi/2}\int_{0}^{\sqrt2} 3(r\cos\theta - r\sin\theta)r\,\mathrm{d}r\,\mathrm{d}\theta $
which can be simplified a bit to
$\displaystyle 3\int_{\pi/4}^{\pi/2}\int_{0}^{\sqrt2} r^2(\cos\theta - \sin\theta)\,\mathrm{d}r\,\mathrm{d}\theta $
To solve it you can go ahead and split the integrand:
$3\int_{\pi/4}^{\pi/2}(\cos\theta - \sin\theta)\,\mathrm{d}\theta\int_{0}^{\sqrt2} r^2\,\mathrm{d}r$
Which are both really trivial integrals to solve.