The perhaps unexpectedly simple reason why you can't evaluate this integral is that it diverges.
You're trying to calculate the potential generated by an infinite charged cylinder from first principles, but this diverges because it's basically the integral over $1/z$. You can get the field using Gauß' law and then integrate it to find a potential, but that potential differs by an infinite additive constant from the integral over the potential of the charge distribution. The situation is similar to that of an infinite charged sheet.
If you want to evaluate integrals instead of using Gauß' law, you can evaluate the integral for the field using the Weierstraß substitution:
$ \begin{align} f'(r) &=\int_{-\infty}^{\infty}\mathrm dz\int_0^{2\pi}\mathrm d\phi\frac{R\cos\phi-r}{\sqrt{z^2+r^2+R^2-2rR\cos\phi}^3} \\ &= 2\int_0^{2\pi}\mathrm d\phi\frac{R\cos\phi-r}{r^2+R^2-2rR\cos\phi} \\ &= 4\int_0^\pi\mathrm d\phi\frac{R\cos\phi-r}{r^2+R^2-2rR\cos\phi} \\ &= 4\int_0^{\infty}\frac{R(1-t^2)/(1+t^2)-r}{r^2+R^2-2rR(1-t^2)/(1+t^2)}\frac{2\mathrm dt}{1+t^2} \\ &= 4\int_{-\infty}^{\infty}\frac{R(1-t^2)-r(1+t^2)}{(r^2+R^2)(1+t^2)-2rR(1-t^2)}\frac{\mathrm dt}{1+t^2} \\ &= -4\int_{-\infty}^{\infty}\frac{r-R+(r+R)t^2}{(r-R)^2+(r+R)^2t^2}\frac{\mathrm dt}{1+t^2} \\ &= -\frac4{r+R} \int_{-\infty}^{\infty}\frac{\lambda+t^2}{\lambda^2+t^2}\frac{\mathrm dt}{1+t^2} \end{align} $
with $\lambda=(r-R)/(r+R)$. We can close the contour with a semicircle at infinity since the integrand decays as $t^{-2}$. The poles are at $\pm\mathrm i$ and $\pm\lambda\mathrm i$. The residue at $\mathrm i$ is
$\frac{\lambda-1}{\lambda^2-1}\frac1{2\mathrm i}=\frac1{\lambda+1}\frac1{2\mathrm i}=\frac{r+R}{4r\mathrm i}\;,$
and the one at $\lambda\mathrm i$ is also
$ \frac{\lambda-\lambda^2}{2\lambda\mathrm i}\frac1{1-\lambda^2}=\frac1{2\mathrm i}\frac1{1+\lambda}=\frac{r+R}{4r\mathrm i}\;. $
For $R\gt r$, we have $\lambda\lt0$, so we need the residue at $-\lambda\mathrm i$, so the contributions cancel and the integral vanishes; whereas for $R\lt r$ we have $\lambda\gt0$, so the contributions add up and the integral is
$ 2\pi\mathrm i\frac{-4}{r+R}2\frac{r+R}{4r\mathrm i}=-\frac{4\pi}r\;. $