Let’s go back to the original polynomial,
$x^3+x^2y-x^2+2xy+y^2-2x-2y\;.\tag{1}$
That $2xy$ looks a lot like the middle term of $(x+y)^2$, and the $-2x-2y$ can certainly be written nicely in terms of $x+y$, so let’s try something along those lines. $(x+y)^2=x^2+2xy+y^2$, and we have the $2xy+y^2$, but instead of $x^2$, we have $-x^2$. In other words, we have $(x+y)^2-2x^2$. That’s not entirely promising, but let’s see where it goes. We can rewrite $(1)$ as
$\begin{align*} (x+y)^2&-\;2x^2-2(x+y)+x^3+x^2y\\\\ &=(x+y)^2-2(x+y)+x^2(x+y-2)\\\\ &=(x+y-2)(x+y)+x^2(x+y-2)\\\\ &=(x+y-2)(x^2+x+y)\;. \end{align*}$
Added: Alternatively, following the hint, interchange the rôles of $x$ and $y$, rewriting $(1)$ as $y^2+(x^2+2x-2)y+(x^3-x^2-2x)\;.\tag{2}$
Remember that linear factors of a polynomial correspond to zeroes. Set $(2)$ equal to $0$ and use the quadratic formula to solve for $y$:
$\begin{align*} &(x^2+2x-2)^2-4(x^3-x^2-2x)\\ &\qquad=x^4+4x^3-8x+4-4x^3+4x^2+8x\\ &\qquad=x^4+4x^2+4\\ &\qquad=(x^2+2)^2\;, \end{align*}$
so $y=\frac{-x^2-2x+2\pm(x^2+2)}2\;,$
and $y=-x^2-x$ or $y=-x+2$. Thus, $(2)$ factors as $\Big(y-(-x^2-x)\Big)\Big(y-(-x+2)\Big)=(y+x^2+x)(y+x-2)\;.$