Let
$$ \ f(x) = \begin{cases} x^{2}\sin(\frac{1}{x}), & \text{if $x \not = 0$} \\ 0, & \text{if $x = 0$} \end{cases} $$
and $$g(x) = \begin{cases} 0, &\text{if } x\not \in \mathbb Q \\ \frac{1}{b}, &\text{if } x = \frac{a}{b} \ \text{ with } a \in \mathbb Z, \ b \in \mathbb Z^{+} \text{ and } \gcd(a,b) = 1 \end{cases}$$ Let $h(x) = f(g(x))$
Determine where $h(x)$ is continuous and differentiable.
Here is my trial.. firstly derive $$ h(x) = \begin{cases} \frac{1}{b^{2}} \sin(b) & \text{where } x = \frac{a}{b}, \\ 0 & \text{where } x \not \in \mathbb Q \end{cases}$$
Then we can show $h(x)$ is not continuous at every $x \in \mathbb Q $ as following: Assume $p = \frac{a}{b}$ then $\ f(p) = f(\frac{a}{b}) = \frac{1}{b^{2}}\sin(b)$, but given any irrational sequence $x_{n}$ converging to p} \$ we have $\lim_{n \to \infty}f(x_{n}) = 0$ since $x_{n}$ is irrational, then we proved the discontinuity in $\mathbb Q$.
But how can I prove that $h$ is continuos for $x \not \in \mathbb Q$ ? Hope someone can help...