In order to show that $f(x,y)$ has a local minimum at $(0,0)$, you need to show that there is an $\epsilon\gt 0$ such that for all $(x,y)$, if $\sqrt{(x-0)^2+(y-0)^2}\lt\epsilon$ (that is, if $(x,y)$ is $\epsilon$-close to $(0,0)$), then $f(0,0)\leq f(x,y)$. In other words: there is an open ball around $(0,0)$ with the property that $f(0,0)$ is the smallest value that $f$ takes in that ball.
So you are not "given an $\epsilon\gt 0$", as you are with limits. Instead, you get to pick the $\epsilon$.
However, you don't get to restrict the values of $x$ and $y$: any value of $x$ and $y$ that falls within $\epsilon$ of $(0,0)$ has to be accounted for. So you don't get to say "Let $|y|=-x$." And you don't get to pick $x$, either.
So, I'm afraid that this is not a valid way to go about it.
Instead, let's think about $f(x,y)$. Both the $x^4$ and the $y^2$ will contribute positive factors; so really we only have a potential problem (i.e., a value of $f$ smaller than $0$) when $x\lt 0$; if $y=0$ the factor $3xy^2$, the only possible source of problems, vanishes, so we just need to worry about the case $y\neq 0$. Then we have $f(x,y) = x^4 + 3xy^2 + y^2 = x^4 + y^2(3x+1)$.
This means that if $3x+1\geq 0$, then $f(x,y)\geq 0 = f(0,0)$, even if $x$ is negative and has absolute value much larger than $y$. And this will be achieved if $x\geq - \frac{1}{3}$. So all you need to do is make sure that your $x$ is larger than $-\frac{1}{3}$ (as you probably noticed, given the value you wanted to assign to $x$).
So: all you need to do is find some $\epsilon$ with the property that if $\sqrt{x^2+y^2}\lt \epsilon$, then you'll know that $x\geq -\frac{1}{3}$. Then you can argue that for that $\epsilon$, you have the property you want (every point $(x,y)$ within $\epsilon$ of $(0,0)$ will have $f(x,y)\geq f(0,0)$). How do you find the $\epsilon$? Draw a picture: $\epsilon$ is the radius of a circle around $(0,0)$; you want every point inside the circle to have $x$-coordinate greater than or equal to $-\frac{1}{3}$. What value(s) can you pick for $\epsilon$? (There is no single answer, since if you have a value that works, any smaller value will also work).