I got stuck on this system of equations. Could you help and tell me how should I approach this problem?
\begin{align*} x+y^2 &= y^3\\ y+x^2 &= x^3 \end{align*}
These are the solutions: \begin{align*} (0, 0); \\((1+\sqrt{5})/2, (1+\sqrt{5})/2); \\((1-\sqrt{5})/2, (1-\sqrt{5})/2); \end{align*}
Here's your equations:
$x+y^2 = y^3$
$y+x^2 = x^3$
I could substitute $x=y^3-y^2$ and get a degree 9 equation in $y$, but I'll try something else.
Looking at these, I notice that if I subtract them, I get something in which everything is divisible by $x-y$.
Subtracting the second from the first, I get
$(x-y)+(y^2-x^2) =y^3-x^3 $
or $(x-y)+(y-x)(y+x) =(y-x)(y^2+xy+x^2) $.
If $y \ne x$, dividing by $y-x$ gives $-1+(y+x) =y^2+xy+x^2 $ or $0 =y^2+y(x-1)+x^2-x+1 $.
Solving this,
$\begin{array}\\ y &=\dfrac{-(x+1)\pm \sqrt{(x-1)^2-4(x^2-x+1)}}{2}\\ &=\dfrac{-(x+1)\pm \sqrt{x^2-2x+1-4x^2+4x-4}}{2}\\ &=\dfrac{-(x+1)\pm \sqrt{-3x^2+2x-3}}{2}\\ &=\dfrac{-(x+1)\pm \sqrt{-2x^2-x^2+2x-1-2}}{2}\\ &=\dfrac{-(x+1)\pm \sqrt{-2x^2-(x-1)^2-2}}{2}\\ \end{array} $
Since the discriminant is negative, there are no real values of $y$.
Therefore the only real solution is $x=y$.
Notice that $x=0 \implies y=0$, hence $(0,0)$ is a trivial solution. We will consider the case where $x \neq 0$.
$$x=y^2(y-1)$$ $$y=x^2(x-1)$$
Suppose on the contrary that $x=1$, then from the second equation $y=0$ which contradicts the first equation $x=0$. Hence $x \neq 1$.
Suppose on the contrary that $x \in (0,1)$,
$$y=x^2(x-1) \in (-1,0)$$
and hence $$x=y^2(y-1)<0$$
which is a contradiction. Hence $x \notin (0,1).$ By symmetry, $y \notin (0,1)$.
Also, notice if $x>1$, then $y=x^2(x-1)>0$, and hence $y>1$.
and if $x<0$ then $y=x^2(x-1)<0$.
$$x=y^2(y-1)$$
$$x^2(x-1)=y$$
Multiply the two equations together, we have
$$x^3(x-1)=y^3(y-1)$$
Consider the function, $$f: (1,\infty) \rightarrow (0,\infty),\text{ where }f(t)= t^3(t-1).$$
We can easily see that this function is increasing and it is an injective function.
Hence, if $x>1$, we have $f(x)=f(y)$ and hence $x=y$.
Similarly, we can consider the function, $$g: (-\infty,0) \rightarrow (0,\infty),\text{ where }g(t)= t^3(t-1).$$
We can easily see that this function is decreasing and it is an injective function.
Hence, if $x<0$, we have $g(x)=g(y)$ and hence $x=y$.
Hence, we always have $x=y$.
$$x+x^2=x^3$$
$$x(1-x-x^2)=0$$
$x=0$ or $1-x-x^2=0$
Since $y=x^3-x^2$, we get $x+(x^3-x^2)^2=(x^3-x^2)^3$ or $$x(x^2-x-1)(x^6-2x^5+2x^4-2x^3+2x^2-x+1)=0$$ and since $x^6-2x^5+2x^4-2x^3+2x^2-x+1=(x^6-2x^5+x^4)+(x^4-2x^3+x^2)+(x^2-x+1)>0$,
we get your answer.
Here is a dynamical systems approach assuming that we only look for real solutions. Consider the map: $$f(x)=x^3-x^2=x^2(x-1)$$ Solving $f(x)=y$ and $f(y)=x$ is equivalent to solving $f(f(x))=x$. We want to show that a solution of the latter (a periodic orbit of period 2) is necessarily a fixed point, i.e. $f(x)=x$ which is trivially solvable.
So assume that $x=f(y), y=f(x)$. We may assume that $x\leq y$.
Now $f$ is strictly monotone increasing on $[1,+\infty)$ so if $1\leq x\leq y$ then $y=f(x)\leq x=f(y)$ implying $x=y$.
Similarly, as $f$ is strictly monotone on $(-\infty,0]$ then $x\leq y\leq 0$ implies $x=y$.
Finally, $f$ maps $(-\infty,1]$ into $(-\infty,0]$ so $x\in (0,1]$ implies $x=f(f(x))\leq 0$ (contradiction). We conclude that $x=y$ as wanted.
Remark: Incidently, the same argument shows that any real orbit of period $p$ must also be a fixed point. So e.g. $x+y^2=y^3$, $y+z^2=z^3$, $z+x^2=x^3$ has the same (real) solutions.
The solution set of the second equation is the cubic parabola $$\gamma:\quad y=x^3-x^2$$ which intersects the symmetry line $y=x$ in the three points $$\left({1-\sqrt{5}\over2},{1-\sqrt{5}\over2}\right),\quad(0,0),\quad\left({1+\sqrt{5}\over2},{1+\sqrt{5}\over2}\right)\ .\tag{1}$$ Otherwise $\gamma$ lies in the interior of the four shaded regions of the following figure (note that for $x>0$ one has $x^3-x^2+x=x(x^2-x+1)>x(x-1)^2>0$, hence $x^3-x^2>-x$):
The solution set of the first equation is the curve $\hat\gamma$ obtained by reflecting $\gamma$ in the symmetry line $y=x$. It contains the three points $(1)$ and is otherwise a subset of the reflected shaded regions. Since the latter are disjoint from the original regions it follows that the intersection $\gamma\cap\hat\gamma$ is given by $(1)$.