For simplicity I'll use $a=[x]$, $b=\{x\}$, $x=a+b$. The equation becomes $$2|a+b-1|=a(a-1)(2a-1)/6+a^2b$$.
First consider the case $a\geq 1$, then $x\geq 1$ We can estimate $$RHS-LHS\geq a(a-1)(2a-1)/6-2(a+b-1)\geq a(a-1)(2a-1)/6-2a=\\ \frac{a(2a^2-3a-11)}{6}$$
Last formula is strictly positive when $a\geq 4$ (Routine calculus or induction exercise). So in this case all possible values of $a$ are $1, 2, 3$.
Similarly when $a\leq 0$, $$RHS-LHS=a(a-1)(2a-1)/6+a^2b-2(1-a-b)\leq a(a-1)(2a-1)/6+a^2+2a=\frac{a(2a^2+3a+13)}{6}$$. Again use calculus or induction to prove last expression is strictly negative when $a\leq -1$. So in this case all possible value of $a$ is $0$.
Now plug in $a=0,1,2,3$ and solve $b$ for each case (Don't forget $0\leq b<1$). $x=a+b$ and if I compute it correctly $x=1$ or $5/2$