For $(a)$, first show that $\cos (n+2)x - 2\cos x\cos(n+1)x + \cos nx = 0$ for all $x$. This follows by almost direct application of the sum rules for $\cos$. Indeed, it might be easier to show if you write it as $\cos(m+1)x + \cos(m-1)x = 2\cos x\cos mx$ where $m=n+1$. The rest of $(a)$ follows with some manipulation. (It's not quite as easy as it looks.)
For $(b)$, you've assumed $c_1$ and $c_2$ are real values. They are not. They are possibly complex functions of $\phi$.
The actual resulting formula should be $I_n(\phi)=\frac{\pi\sin n\phi}{\sin \phi}$
One other thing to note is that if $\sin\phi = 0$ then $x_1=x_2$, so you have to adjust your general formula for the recurrence relationship to the case where your recurrence polynomial has repeated roots. Then $x_1=x_2=x=\pm 1$. If $x=+1$ then $I_n = c_0+nc_1$ and we get that $I_n = n\pi$. If $x=-1$, then $c_0=0$ and $c_1=-\pi$ and $I_n=(-1)^{n+1}\pi n$. This is actually just the limit - it is the value which makes $I_n(\phi)$ continuous at these values.
In the calculation for $(a)$, when you do the substitution listed at the top in the expression $\frac{\cos(n+2)\theta - \cos(n+2)\phi}{\cos\theta-\cos\phi}$ you get:
$\frac{2\cos \theta \cos(n+1)\theta - \cos n\theta - (2\cos\phi\cos(n+1)\phi -\cos n\phi)}{\cos\theta-\cos\phi}$
The trick is to write $\cos \theta = (\cos\theta - \cos\phi) + \cos\phi$. Substituting, we get:
$2\cos(n+1)\theta + 2\cos\phi\frac{\cos(n+1)\theta - \cos(n+1)\phi}{\cos\theta-\cos\phi} - \frac{\cos n\theta -\cos n\phi}{\cos\theta -\cos\phi}$
Then integrating, you get $I_{n+2}(\phi)=\int_{0}^\pi 2\cos(n+1)\theta\ d\theta + 2\cos\phi I_{n+1}(\phi) - I_n(\phi)$
But $\int_{0}^\pi 2\cos(n+1)\theta\ d\theta=0$.
So you are done.