The proof of (a) is by induction. There is not quite increase always, since $x_1=0$. But we can compute a step further, and show that $x_2\gt x_1$ and $y_2\gt y_1$.
Assume that $k \ge 1$, and $x_{k+1}\gt x_k$ and $y_{k+1}\gt y_k$. We prove that $x_{k+2}\gt y_{k+1}$ and $y_{k+2}\gt y_{k+1}$.
This is relatively easy. For ease of typing, we deal with the $y$'s. We have $y_{k+2}=\frac{x_{k+1}+y_{k+1}+1}{3}$. But by the induction assumption, we have $x_{k+1}\gt x_k$ and $y_{k+1}\gt y_k$. It follows that $y_{k+2}=\frac{x_{k+1}+y_{k+1}}{3}\gt \frac{x_k+y_k+1}{3}=y_{k+1}.$
The argument for $x_{k+2}$ is very similar.
For boundedness, we prove by induction that $x_n \lt 1$ and $y_n \lt 1$ for all $n$. The two inuction steps are quite straightforward.
Because our two sequences $(x_n)$ and $(y_n)$ are non-decreasing and bounded above, they each have a limit.
As to the limits, if $x$ is the limit of the $x_n$, and $y$ is the limit of the $y_n$, we have $x=\sqrt{\frac{x^2+2y^2}{4}}$ and $y=\frac{x+y+1}{3}$. If we square both sides of the first equation, we obtain an expression of $y$ in terms of $x$. Substituting for $x$ in the second equation, we find $y$. Then we compute $x$.