When you set out to differentiate a function, look at the last operation that you would perform in computing a value of the function. In the case of $\sqrt t(t-1)$, that operation is multiplying $\sqrt t$ and $t-1$. That means that this function is (at the top level) a product and must be differentiated using the product rule:
$g'(t)=\left[\sqrt t(t-1)\right]'=\left[\sqrt t\right][t-1]'+\left[\sqrt t\right]'(t-1)\;.$
Now $[t-1]'=1-0=1$, exactly as you computed it in the question, but to differentiate $\sqrt t$, you’ll want to convert it to a power, $t^{1/2}$, and use the power rule:
$\left[\sqrt t\right]'=\left[t^{1/2}\right]'=\frac12t^{-1/2}\;.$
Now put the pieces together:
$\begin{align*} g'(t)&=\sqrt t\cdot1+\frac12t^{-1/2}(t-1)\\ &=t^{1/2}+\frac12\left(t^{1/2}-t^{-1/2}\right)\\ &=\frac32t^{1/2}-\frac12t^{-1/2}\\ &=\frac12\left(3t^{1/2}-t^{-1/2}\right)\;. \end{align*}$
Some instructors might prefer that you rewrite that as $\frac12\left(3t^{1/2}-t^{-1/2}\right)=\frac12\left(3\sqrt t-\frac1{\sqrt t}\right)=\frac{3t-1}{2\sqrt t}\;.$
Second Addition: I should have noticed that I was actually doing this the hard way. The easy way is to convert $\sqrt t$ to $t^{1/2}$ immediately and multiply out to get $g(t)=t^{1/2}(t-1)=t^{3/2}-t^{1/2}$. Now you can apply the power and sum rules to get $g'(t)=\frac32t^{1/2}-\frac12t^{-1/2}$ directly and simplify from there.
Added: If by chance your function was intended to be $g(t)=\sqrt{t(t-1)}$, the same principle applies, but the details are different. Now the last thing that you’d do in computing a value of $g$ is take the square root, so at the top level this is a power: $g(t)=\Big(t(t-1)\Big)^{1/2}\;.$
Thus, you’ll need to apply the power rule, combined, of course, with the chain rule:
$g'(t)=\frac12\Big(t(t-1)\Big)^{-1/2}\Big[t(t-1)\Big]'\;.$
You could use the power rule to do that last differentiation, but it’s much easier to multiply out and differentiate $t^2-t$ instead:
$g'(t)=\frac12\Big(t(t-1)\Big)^{-1/2}(2t-1)=\frac{2t-1}{2\sqrt{t^2-t}}\;.$