Here are some extended hints.
Since $f(1/4)=1/4$ and $f(1/2)=(1/2)$, the function cannot be decreasing. Now look at two irrational numbers in $(0,1)$, say $\pi/5$ and $\pi/4$; what are $f(\pi/5)$ and $f(\pi/6)$? Can the function $f$ be monotonically increasing on $(0,1)$?
To show that $f$ is a bijection, you must show that it’s both injective (one-to-one) and surjective (onto).
To see that $f$ is injective, ask whether it’s possible to have two different numbers $x,y\in(0,1)$ with $f(x)=f(y)$. Do it in three cases.
- If $x,y\in\Bbb Q$, and $x\ne y$, can $f(x)=f(y)$?
- If $x,y\in\Bbb R\setminus\Bbb Q$, and $x\ne y$, can $f(x)=f(y)$?
- If $x\in\Bbb Q$ and $y\in\Bbb R\setminus\Bbb Q$, can $f(x)=f(y)$?
To see that $x$ is surjective, show that no matter what $y\in(0,1)$ you choose, there is an $x\in(0,1)$ such that $f(x)=y$. Here again it makes sense to look at cases.
- What should $x$ be if $y\in\Bbb Q$?
- What should $x$ be if $y\in\Bbb R\setminus\Bbb Q$?
Added: These ideas are easily modified to show that $f$ cannot be monotonic on any subinterval of its domain or range and that its restriction to any subinterval of its domain or preimage of any subinterval of its range must be a bijection. For the non-monotonicity, use the fact that any non-trivial interval of real numbers contains both rational and irrational numbers, instead of using specific ones as I did above. Injectivity of $f$ on a subinterval follows immediately from injectivity of $f$ on all of $(0,1)$.