Here's one way to find the indefinite integral.
Write everything in terms of $\sin$ and $\cos$. If one of those functions is raised to an odd positive power, say $\cos$, factor one term of $\cos$ out and use the Pythagorean identity to write the other factor in terms of $\sin$. Then use a $u$-substitution with $u=\sin x$:
$\eqalign{ \int\cot^5 x\,\csc^3 x\,dx&= \int {\cos^5 x\over \sin^5 x}{1\over \sin^3 x}\,dx\cr &=\int {\cos x \cdot(\cos^2 x)^2\over \sin^8 x} \,dx\cr &=\int {\cos x \cdot(1-\sin^2 x)^2\over \sin^8 x} \,dx\cr &\buildrel{u=\sin x}\over=\int { (1-u^2)^2\over u^8 } \,du\cr &=\int { 1-2u^2+u^4 \over u^8 } \,du\cr &=\int {( u^{-8}-2u^{-6}+u^{-4} )} \,du\cr &= {{ -u^{-7}\over7}+2{u^{-5}\over 5}-{u^{-3}\over3} } +C\cr &= {{ -\sin^{-7} x\over7}+2{\sin^{-5}x\over 5}-{\sin^{-3}x\over3} } +C.\cr } $