The following approach is more lengthy than necessary, but uses only basic ideas. We show how to find the cumulative distribution function of $Z$. Then differentiation gives the density.
The joint density function of $X$ and $Y$ is $2$ in the interior of the triangle with corners $(0,0)$, $(1,0)$, and $(1,1)$, and $0$ elsewhere.
We want to find $\Pr(Z \le z)$. The only interesting part is for $0\lt z\le 1$.
To find this probability, we calculate $\iint_D f(x,y)\,dx\,dy,$ where $f(x,y)$ is the joint density, and $D$ is the region on or below the curve $xy^3=z$. For fixed $z$, draw that curve. It will meet the line $y=x$ at $x=z^{1/4}$.
Now our integral is the integral over the part of $D$ with $x\lt z^{1/4}$ (this is a triangle), plus the integral over the part of $D$ with $x$ going from $z^{1/4}$ to $1$.
The first integral is just $2$ times the area of the triangle with base $z^{1/4}$ and height $z^{1/4}$.
The second integral is more complicated. Integrate first with respect to $y$, with $y$ going from $0$ to $(z/x)^{1/3}$. This integral is simply $2(z/x)^{1/3}$. Then integrate from $x=z^{1/4}$ to $x=1$.