I was wondering if there was a substitution formula to solve integrals of this form:
\int f(g(x))g''(x)dx
I was wondering if there was a substitution formula to solve integrals of this form:
\int f(g(x))g''(x)dx
No, not a nice one, anyway. It is worthwhile, I think, to point out that integration rules, such as the usual substitution rule, do not always "solve" ("evaluate" is the proper term) the given integral. The usual substitution rule, for instance, only transforms the integral into another integral which may or may not be easily handled.
Of course, if the antiderivative of $f$ is known, then the usual substitution rule will allow you to evaluate integrals of the form \int f\bigl(g(x)\bigr)g'(x)\,dx. I don't think a formula of the type you seek would be very useful, as it can't handle all cases when an antiderivative of the "outer function" is known: consider $\int \sin(x^2)\cdot 2\,dx$. This can't be expressed in an elementary way.
What you can do depends a lot on the form of $g(x)$. I doubt very much that there is a general solution.
However there's two steps that I'd look at doing: (These are only valid if $g(x)$ is smooth and monotonic)
First an integration buy parts, then a substitution.
Taking $u = f(g(x))$ and dv = g''(x)\;dx we'd get that:
\int f(g(x))g''(x)\;dx = f(g(x))g'(x) - \int f'(g(x))g'(x)^2\;dx
Now making the substitution $u = g(x)$ (which will only be valid for some $g(x)$), so that du = g'(x)\;dx, we get:
\int f(g(x))g''(x)\;dx = f(g(x))g'(x) - \int f'(u) g'( g^{-1}(u) )\;du
Whether or not this is an improvement will depend on what g'(g^{-1}(u)) is like.