Usually this problem is seen with $g$ being some trigonometric function with the specical condition $\displaystyle \int^1_0 g(x) dx = 0 \ ;$ in this case the result is easy to discover. Intuitively, for large enough $n$ the oscillations become extremely rapid and since piece-wise constant functions uniformly approximate continuous functions arbitrarily well, the areas cancel out. So it is natural to prove this special case first:
Suppose $\displaystyle \int^1_0 g(x) dx =0. $ Let $G(x) =\displaystyle \int^x_0 g(t) dt.$ Since $g$ is $1$-periodic and $\displaystyle \int^1_0 g(x) dx=0,$ we have that $G$ is bounded on $ \mathbb{R}.$
If $f \in C^1( [0,1],\mathbb{R} )$ then integration by parts gives $ \int^1_0 f(x) g(nx) dx = \frac{ f(1)G(n) - f(0)G(0) }{n} - \frac{1}{n} \int^1_0 f'(x) G(nx) dx \to 0$ as required. As note that if $\displaystyle \int^1_0 g(x) dx \neq 0 $ then the limit is not $0$ for arbitrary $f\in C[0,1].$
If $f$ is continuous but not nessicarily $C^1$ then by the Stone-Weierstrass Theorem we can find a continuously differentiable $h: [0,1] \to \mathbb{R}$ such that $\displaystyle \int^1_0 |f(x)-h(x)| dx$ is arbitrarily small. Then since $ \biggr| \int^1_0 \left( f(x) - h(x) \right) g(nx) dx \biggr| \leq \sup_{x\in [0,1]} |g(x)| \int^1_0 | f(x)-h(x)| dx$ is also arbitrarily small, we have $\lim_{n\to\infty} \int^1_0 f(x) g(nx) dx = \lim_{n\to \infty} \int^1_0 h(x)g(nx) dx =0.$
How can one discover the general result? In the language of linear algebra, what we have shown is that $\displaystyle \int^1_0 f(x) g(nx) dx \to 0$ if and only if $g\in \ker L$, where $L:C[0,1]\to C[0,1]$ is the linear operator defined by $L(g) =\displaystyle \int^1_0 g(x) dx.$ So naturally, we want to see what this kernel looks like. We can see that $L^2=L$ so then we find that $ \ker L = \{ g- Lg : g\in C[0,1] \}.$ So the special case is true if we use $g-Lg,$ and rewriting that gives precisely the general result. We could compress this into a more mysterious conclusion:
Now, for arbitrary $ \displaystyle \int^1_0 g(x) dx$, the function $ \hat{g}(x)=g(x) -\displaystyle \int^1_0 g(x) dx $ is a continuous $1$-periodic function with $\displaystyle \int^1_0 \hat{g}(x) dx=0$ and applying the previously developed result to it gives the full result.