The first problem is that you substituted $x=\frac12(1-a)$ into $f(x)$ incorrectly; the second (and more important) problem is that you need to define a new function whose independent variable is $a$. Specifically, let $g(a)$ be the maximum attained by the function $f(x)=ax(1-x-a)$; you want to find the value of $a$ that maximizes $g(a)$. Substituting $x=\frac12(1-a)$ into $f(x)$, we find that
$\begin{align*} g(a)&=a\left(\frac{1-a}2\right)\left(1-\frac{1-a}2-a\right)\\ &=\frac{a}4(1-a)\big(2-(1-a)-2a\big)\\ &=\frac{a}4(1-a)(1-a)\\ &=\frac14\left(a-2a^2+a^3\right)\;. \end{align*}$
Now $g'(a)=\frac14\left(1-4a+3a^2\right)$. Setting this equal to $0$, we have $3a^2-4a+1=0$. To solve for $a$ you can either use the quadratic formula or notice that $3a^2-4a+1=(3a-1)(a-1)$; either way, you find that $g'(a)=0$ for $a=1$ and $a=\frac13$. By analyzing the sign of $g'(a)$ or by using the second derivative test you can check that $g(a)$ has a local maximum (of $\frac1{27}$) at $a=\frac13$ and a local minimum (of $0$) at $a=1$.
However, a quick check of the graph of $g(a)$ will show you that it increases without bound as $a\to\infty$, and this is also clear algebraically: as $a\to\infty$, $1-a\to-\infty$, so $\frac14a(1-a)^2\to\infty$. Thus, you can make the maximum of $f(x)$ as large as you want by choosing $a$ large enough.