There's probably no standard way to solve such a problem, since integration is art; this is just how I would approach this particular problem.
Simplifying and dropping constant factors leads to
$$\int\frac{2xf(x+1)-(x+1)}{x^2(-5x-1)^{3/5}}(x+1)^2\mathrm dx\;.$$
The problem is that we have two different factors in the denominator. If we can choose $f$ such that it gets rid of one of them while leaving a polynomial in the numerator, we can make a linear transformation that turns the other one into a power through which we can divide the numerator. Thus we'd like to have
$$2xf(x+1)-(x+1)=x^2p(x)$$
with $p(x)$ a polynomial. Substituting $t=x+1$ and solving for $f$ yields
$$f(t)=\frac12\left((t-1)p(t-1)+\frac{t}{t-1}\right)\;.$$
This has a pole at $t=1$. You've specified a value for $t=0$ and the integrand has a singularity at $t=4/5$ and is undefined for $t\gt4/5$ since the radicand becomes negative, so I assume that you wouldn't mind a pole at $t=1$. If so, we just have to choose $p$ to fulfill the conditions on $f$. The condition $f(0)=0$ implies that $p$ vanishes at $t=0$, and thus $p(t-1)=tq(t-1)$. Other than that, we're free to choose $p$ such that $f$ is increasing. The second term in $f$ is decreasing, so the first term has to compensate for it. The first term is proportional to $t(t-1)q(t-1)$, and we must have $q(-1)\le-1$ for $f$ to be increasing at $t=0$. Since $-t(t-1)$ is decreasing for $t\gt1/2$, $-q$ needs to increase sufficiently in that range to compensate the decrease. We can't have further factors of $t$, but we can add any number of factors of $t+1$. With $q(t-1)=-2(t+1)^n$, we have
$$f(t)=-t(t-1)(t+1)^n+\frac12\frac t{t-1}\;.$$
Trial and error shows that $n=9$ is enough to make $f$ increasing up to $t=4/5$. (Here's a plot.) This corresponds to
$$p(x)=-2(x+1)(x+2)^9\;,$$
so the integral becomes (up to constant factors)
$$\int \frac{(x+1)^3(x+2)^9}{(-5x-1)^{3/5}}\mathrm dx\;.$$
Now the substitution $u=-5x-1$ leads to a trivial integral.