You draw a picture in the y-z plane, and you rotate it by 45 degrees and blow up by a factor of $\sqrt{2}$. This gives you a triangle region, where the vertices are determined to be the image of the old vertices under the rotation: (x,x),(a,x),(a,a). The images in the $\beta,\gamma$ plane are $(2x,0),(a+x,a-x),(2a,0)$.
But I understand that you want a formal algorithm, so that you can do it mechanically. First, you make a one-variable transformation, where you trade in z for $\gamma= z-y$. If you really insist on doing the rotation (it's not the right thing), you can then trade in $y$ for $\beta$, using $\beta= 2y + \gamma$.
The answer for the first shift is that the z range is from y to a, so translating z by -y goes from 0 to a-y.
$\int_0^a dx f(x) \int_x^a dy \int_0^{a-y} d\gamma g(\gamma) $
To complete the mechanical change of variable, introduce an indicator function of one dimension $\phi(x)$, which is 1 for $x>0$ and 0 for $x<0$. Then you write the integral as follows:
$ \int_0^a dx f(x) \int_x^a dy \int_0^\infty \phi(a-y-\gamma) g(\gamma) d\gamma$
Rearrange the order of the y and $\gamma$ integrals and perform the y integral explicitly (using the fact that the indefinite integral of $\phi(x)$ is $x\phi(x)$) to get
$\int_0^a dx f(x) \int_0^\infty (a-x-\gamma)\phi(a-x-\gamma)) g(\gamma) d\gamma $
The $\phi$ function now gives you the new domain for $\gamma$
$ \int_0^a dx f(x) \int_0^{a-x} (a-x-\gamma)g(\gamma)$
and this is the best form. Using explicit indicator functions ($\phi$) this way is useful in many cases for getting explicit answers in closed form. If you really want to do the transformation of variables the way you said it, you introduce the indicator functions for the region, and then transform the region and find the new domain. This is just the same as drawing the triangle and rotating it.