I found this problem a while ago on AoPS.
Let $k$ be a squarefree positive integer.
Find $\inf_{n \in \Bbb{Z}_+^*} n\{n\sqrt{k}\,\}$, where $\{\cdot\}$ denotes the fractional part.
I tried to find a solution, but it is still unsolved. My guess for the infimum is $0$, and I tried proving the existence of a sequence $(n_i)$ such that $\displaystyle \{n_i \sqrt{k}\} < \frac{1}{n_i^2}$. Denote $p_i$ the integer part of $n_i\sqrt{k}$. Then we have $\displaystyle p_i
The first inequality must be strict, and the best case would be $p_i^2-kn_i^2=-1$, which is a Pell equation, solvable for some squarefree $k$, but not for all squarefree $k$.
Do you have some ideas of what to do next?