EDIT: Expanding on J. M.'s advice.
From $\int_0^x {f(t)dt} = f^2 (x)$, we get $f(x) = \frac{d}{{dx}}f^2 (x)$ (using that $f$ is continuous). From this, one wants to write f(x)=2f(x)f'(x). In general, however, differentiability of $f^2$ does not imply that of $f$ (consider, for example, the functions $|x-1|$ and $(x-1)^2$, $x \geq 0$; the former is not differentiable at $x=1$), and we are not given that $f$ is differentiable. Nevertheless, in our case, for any $x > 0$ we have $ f(x) = \frac{d}{{dx}}f^2 (x) = \mathop {\lim }\limits_{h \to 0} \frac{{f^2 (x + h) - f^2 (x)}}{h} = \mathop {\lim }\limits_{h \to 0} \frac{{[f(x + h) + f(x)][f(x + h) - f(x)]}}{h}, $ leading to $ 1 = \mathop {\lim }\limits_{h \to 0} \bigg[\frac{{f(x + h) + f(x)}}{{f(x)}}\frac{{f(x + h) - f(x)}}{h}\bigg], $ where we have used the assumption $f(x)>0$. Since $f$ is continuous at $x$, it thus follows that $ 1 = 2\mathop {\lim }\limits_{h \to 0} \frac{{f(x + h) - f(x)}}{h}, $ and hence the conclusion that f'(x) = \frac{1}{2}. (The rest is straightforward.)
Original answer (see first comment below).
If $f$ is continuous on $[0,\infty)$, then, on the one hand, $ \mathop {\lim }\limits_{x \to 0^ + } \int_0^x {f(t)dt} = 0, $ and, on the other hand, $ \mathop {\lim }\limits_{x \to 0^ + } f^2 (x) = f^2 (0). $ So the condition $\int_0^x {f(t)dt} = f^2 (x)$ implies that $f^2 (0) = 0$, hence also $f(0)=0$.