You are on the correct path, it is very similar to the previous one. As you noted, we have $\sum_{n\leq x}\frac{\phi(n)}{n^{2}}=\sum_{n\leq x}\frac{1}{n}\sum_{d|n}\frac{\mu(d)}{d}.$ Switching the order of summation yields $\sum_{d\leq x}\frac{\mu(d)}{d}\sum_{r\leq\frac{x}{d}}\frac{1}{rd}=\sum_{d\leq x}\frac{\mu(d)}{d^{2}}\sum_{r\leq \frac{x}{d}}\frac{1}{r}=\sum_{d\leq x}\frac{\mu(d)}{d^{2}}H_{\left[\frac{x}{d}\right]}.$
Where $H_m$ is the $m^{th}$ harmonic number. Now this is possible to evaluate since $\sum_{n\leq x}\frac{1}{n}=\log x +\gamma+O\left(\frac{1}{x}\right).$ We then get $\sum_{d\leq x}\frac{\phi(d)}{d^2}=\sum_{d\leq x}\frac{\mu(d)}{d^{2}}\left(\log\frac{x}{d}+\gamma+O\left(\frac{d}{x}\right)\right).$
We may rewrite this sum as
$\log x\sum_{d\leq x}\frac{\mu(d)}{d^{2}}-\sum_{d\leq x}\frac{\mu(d)\log d}{d^{2}}+\gamma\sum_{d\leq x}\frac{\mu(d)}{d^{2}}+O\left(\frac{1}{x}\sum_{d\leq x}\frac{|\mu(d)|}{d}\right)$
Since $\sum_{d> x}\frac{\mu(d)\log d}{d^2}=O(\log x/x)$, and $\sum_{d\leq x}\frac{\mu (d)}{d}=O(\log x)$, we see that
$\sum_{n\leq x}\frac{\phi(n)}{n^2}=\log x \sum_{d=1}^\infty \frac{\mu(d)}{d^2}+\gamma \sum_{d=1}^\infty \frac{\mu(d)}{d^2}-\sum_{d=1}^\infty \frac{\mu(d)\log d}{d^2}+O\left(\frac{\log x}{x}\right).$
Looking at these series, $\sum_{d=1}^\infty \frac{\mu(d)}{d^2}=\frac{1}{\zeta(2)}$ and $\sum_{d=1}^\infty \frac{\mu(d)\log d}{d^2}=\frac{\zeta^{'}(2)}{\zeta(2)^2}$, and so
$\sum_{n\leq x}\frac{\phi(n)}{n^2}=\frac{\log x}{\zeta(2)}+\frac{\gamma}{\zeta(2)} -\frac{\zeta^{'}(2)}{\zeta(2)^2}+O\left(\frac{\log x}{x}\right).$
Hope that helps,
Edit: Fixed error with the constant term. I originally missed the $-\log d$ term.
Edit 2: Changed the bound $\sum_{d> x}\frac{\mu(d)\log d}{d^2}=O(1/x)$ to $\sum_{d> x}\frac{\mu(d)\log d}{d^2}=O(\log x/x).$ While the previous one is also true, it requires the prime number theorem. For this result the $\log x$ does not change anything, and there is a much simpler proof. See this answer.