Let us use your formula. First deal with the denominator. What you called $D(X)$ and $D(Y)$ are the standard deviations of $X$ and of $Y$ respectively.
The standard deviation is the square root of the variance, and the variance of a random variable $W$ is $E(W-E(W))^2$.
For $X$, we have $E(X)=0$, so $X-E(X)=X$. So we want $E(X^2)$. When $X=-1$, we have $X^2=1$. When $X=1$ we also have $X^2=1$. So $X^2=1$ with probability $1/4+1/4=1/2$, and $X^2=0$ with probability $1/2$. It follows that $E(X^2)=1/2$, so $D(X)=\sqrt{1/2}$.
Similarly, to find the variance of $Y$, we find $E(Y-E(Y))^2$. Here $E(Y)=1/2$, so we want $E((Y-1/2)^2)$. Now $Y-1/2$ is $-1/2$ with probability $1/2$, and $1/2$ with probability $1/2$. So $(Y-1/2)^2=1/4$ with probability $1$, and therefore has expectation $1/4$. This is the variance of $Y$. Thus $D(Y)=\sqrt{1/4}$.
Now we calculate the numerator, $E((X-E(X))(Y-E(Y))$. We have $X-E(X)=X$ and $Y-E(Y)=Y-1/2$. So we want to find $E(X(Y-1/2))$. That is the stage you were at.
When $X=-1$, we have $Y=X^2=1$, so $X(Y-1/2)=(-1)(1/2)=-1/2$. This happens with probability $1/4$.
When $X=0$, of course $X(Y-1/2)=0$. This happens with probability $1/2$.
When $X=1$, we have $Y=X^2=1$, and therefore $X(Y-1/2)=1/2$. This happens with probability $1/4$.
Now $E(X(Y-1/2))$ is easy to calculate. It is $0$.
Divide by $D(X)D(Y)$. We get $0$. All this work for nothing!
The point of this problem: The random variables $X$ and $X^2$, that is, $X$ and $Y$, are obviously not independent. To see this, note for example that the probability that $X=1$ and $Y=0$ is obviously $0$, so it is not $\Pr(X=1)\Pr(Y=0)$.
But $X$ and $Y$ are uncorrelated (they have correlation coefficient $0$). So the point of the problem is to show you that uncorrelated does not imply independent.
Remark: There are efficient shortcuts for computing covariance and variance. For example, the covariance of $X$ and $Y$ turns out to be $E(XY)-E(X)E(Y)$, which is usually easier to deal with than the expression you quoted. In our case, the situation was so simple that not much added work is involved.
Similarly, the variance of $W$ is $E(W^2)-(E(W))^2$. This is usually easier to compute than $E((W-E(W))^2)$.