You can solve the given integral equation using the Mellin transform techniques. If you take the Mellin transform to both sides of the equation w.r.t $x$, you get
$$ U(s)=U(s)U(s+1)\implies U(s)(1-U(s+1))=0$$
$$\implies U(s)=0\quad \mathrm{or}\quad U(s+1)=1. $$
Taking the inverse mellin transform of the above yields the two solutions
$$ u(x) = 0 \quad \mathrm{or} \quad u(x) = \frac{\delta(x-1)}{x}. $$
Deriving the Mellin Transform of the equation: the Mellin transform of afunction $(x)$ is given by
$$ F(s) = \int_{0}^{\infty}x^{s-1}f(x)dx. $$
Taking the Mellin transform of the equation
$$ \begin{eqnarray*}
u \left( x \right) & = & \int_0^{\infty} u \left( t \right) u \left(
\frac{x}{t} \right) dt
\end{eqnarray*}, $$
yields
$$ \int_{0}^{\infty}x^{s-1}u(x)dx = \int_{0}^{\infty}x^{s-1} \int_0^{\infty} u \left( t \right) u \left(\frac{x}{t} \right) dt\, dx $$
$$ \implies U(s)= \int_{0}^{\infty}u(t) \int_0^{\infty} x^{s-1}u \left(\frac{x}{t} \right) {d} x\, dt $$
Using the change of variables $x=ty$ for the inner integral, we have
$$ \implies U(s)= \int_{0}^{\infty}u(t)t^s dt\int_0^{\infty} x^{s-1}u \left(y\right) dy = U(s+1)U(s).$$