After having a bref look at the method that you intend to use, I noticed that this method is valuable for a certain form of ODE. This is not the case of the ODE :
$$3xy + y^2 + (x^2 + xy) y' = 0$$
where $p$ and $q$ are not functions of $x$ only.
We have to use a more general form of the method of integrating factors. The ODE can be presented as:
$$(3xy+y^2)dx+(x^2+xy)dy=0$$
The aim it to find an integrating factor $\mu(x,y)$ in order to obtain a total derivative of a function $F(x,y)$ to be determined :
$$\mu(x,y)\left((3xy+y^2)dx+(x^2+xy)dy \right)=dF(x,y)$$
so that, the ODE will become $dF(x,y)=0 \quad\to\quad F(x,y)=C$
$$dF=\frac{\partial F}{\partial x}dx+\frac{\partial F}{\partial y}dy \quad\to\quad \begin{cases}
\frac{\partial F}{\partial x}=\mu(x,y)(3xy+y^2)\\
\frac{\partial F}{\partial y}=\mu(x,y)(x^2+xy)
\end{cases}$$
$$\frac{\partial^2 F}{\partial x\partial y}=\frac{\partial }{\partial x}\mu(x,y)(x^2+xy)=\frac{\partial }{\partial y}\mu(x,y)(3xy+y^2)$$
$$(x^2+xy)\frac{\partial \mu(x,y) }{\partial x}+(2x+y)\mu(x,y)=(3xy+y^2)\frac{\partial \mu(x,y) }{\partial y}+(3x+2y)\mu(x,y)$$
At this point, the general process could become very complicated. Since the problem is probably a textbook case, we can suppose that $\mu(x,y)$ is a very simple function. We try more simple forms like $\mu(x)$ or $\mu(y)$ or more complicated if not adequate.
For example, with the variable $x$ only
$$(x^2+xy)\frac{d \mu(x) }{d x}+(2x+y)\mu(x)=(3x+2y)\mu(x)$$
This equation must contains functions of $x$ only $\quad\to\quad \begin{cases}
x\frac{d \mu }{d x}+\mu=2\mu \\
x^2\frac{d \mu }{d x}+2x\mu=3x\mu
\end{cases} \quad\to\quad \mu(x)=x$
We are not looking for the general solution for $\mu(x)$ : only any one particular solution is enough. So, $\mu=x$ is very well, because simple.
$$x\left((3xy+y^2)dx+(x^2+xy)dy \right)=dF(x,y)$$
$$(3x^2y+xy^2)dx+(x^3+x^2y)dy=dF(x,y)=0$$
$$d(x^3y+\frac{1}{2}x^2y^2)=dF(x,y)=0$$
$$F(x,y)=x^3y+\frac{1}{2}x^2y^2=C$$
$$x^2y^2+2x^3y-2C=0$$
$$y=\frac {-x^3 \pm \sqrt{x^6 +2Cx^2}}{x^2}=\frac {-x^2 \pm \sqrt{x^4 +C'}}{x}$$
IN ADDITION :
If we don't want to make an assumption about a simplified form for $\mu(x,y)$ we have to continue from the above equation :
$$(x^2+xy)\frac{\partial \mu(x,y) }{\partial x}+(2x+y)\mu(x,y)=(3xy+y^2)\frac{\partial \mu(x,y) }{\partial y}+(3x+2y)\mu(x,y)$$
$$(x^2+xy)\frac{\partial \mu(x,y) }{\partial x}-(3xy+y^2)\frac{\partial \mu(x,y) }{\partial y}=(x+y)\mu(x,y)$$
Solving this PDE thanks to the method of characteristics starts from the set of ODEs for the characteristics curves :
$$\frac{dx}{x^2+xy}=\frac{dy}{-(3xy+y^2)}=\frac{d\mu}{(x+y)\mu}$$
With the first ODE : $\frac{dx}{x^2+xy}=\frac{dy}{-(3xy+y^2)}$ not surprisingly we come back to the first beginning : $(3xy+y^2)dx+(x^2+xy)dy=0$. This is of no interest.
With the second ODE : $\quad\frac{dx}{x^2+xy}=\frac{d\mu}{(x+y)\mu}\quad\to\quad
\frac{dx}{(x+y)x}=\frac{d\mu}{(x+y)\mu}\quad$ it is obvious that $\quad \mu=x\quad$ is a solution. We are not looking for all solutions for $\mu$. Only one (any one) is sufficient. This one $\mu=x$ is what we where looking for. The end of the calculus was already shown above.
NOTE :
Some particular forms for the integrating factor are ready-made, so avoiding a long calculus. For example , see the cases (11), (12), (13) in :
http://mathworld.wolfram.com/OrdinaryDifferentialEquation.html
