I would distinguish between capital $X$ and $Y$ and lower-case $x$ and $y$, the former being the random variables, and the latter being the variables used in expressions like $f(x,y)$ and in $\int\cdots\cdots\,dx$, etc.
Then we have $ E(X E(Y\mid X)) = \int_0^1 x E(Y\mid X=x) \, f(x) \, dx. $
More generally, for any function $g$, $ E(g(X)) = \int_0^1 g(x) f(x) \, dx. $
Throughout, you should carefully note where I've put capital $X$ and $Y$, and where I've put lower-case $x$ and $y$.
begin quote
However, in another example I have $ E(Y|X) = \frac 12 x. $
Then my professor has $ E(XE(Y|X)) = E(\frac 12 x^2) $ I don't understand why in one case you multiply by $x$ and in the other by $f(x)$
end quote
Again, being careful about capital and lower case, I'd write:
However, in another example I have $ E(Y\mid X) = \frac 12 X $ (with a capital $X$, since this is a random variable).
$ E(XE(Y\mid X)) = E\left(\frac 12 X^2\right) $
After that, you can write $ E\left(\frac 1 2 X^2 \right) = \int_0^1 \left(\frac12 x^2 \right) f(x)\, dx. $