I have been given the following matrix to solve using mixed strategies:
$$ \begin{matrix} 1,2 & 0,0 \\ 0,0 & 2,1 \\ \end{matrix} $$
Given that p and q are the respective probabilities of player 1 and 2, the expected payoff of player 1 i have worked out to be:
payoff_1 = pq + 2(1-p)(1-q) where p and q are the respective probabilities of player 1 and 2. After first order conditions i get
3q - 2 = 0
and
3p - 2 = 0
and i know that the answer is p = 1/3 and q = 2/3, so what am i doing wrong?
The expected payoffs for player 1 are
Alternative 1 (first row): $1q+0\cdot (1-q)$
Alternative 2 (second row): $0q+2\cdot (1-q)$
They have to be equal.
$1q+0\cdot (1-q)=0q+2\cdot (1-q)$
$3q=2\Rightarrow q=\frac23$
Similar for player 2
Alternative 1 (first column): $2p+0\cdot (1-p)$
Alternative 2 (second column): $0p+1\cdot (1-p)$
$2p+0\cdot (1-p)=0p+1\cdot (1-p)$
$3p=1\Rightarrow p=\frac13$
callculus answer is correct and that's how you typically solve those simple 2x2 games. However, I thought I will provide another solution to this question which uses the approach you tried.
Notation: In what follows Player 1 is the row player, Player 2 is the column player.
Let $p$ be the probability with which player 1 plays Top row and $q$ be the probability with which player 2 plays Left column.
Then the expected payoff to player 1 from following a strategy (p,1-p) given that his opponent follows a strategy (q,1-q) is $$ pq + 2(1-p)(1-q)$$ while the expected payoff to player 2 from following a strategy (q,1-q) given that his opponent follows a strategy (p,1-p) is $$ qp + 2(1-q)(1-p) $$
Now, we want to find player 1's best response to player 2's strategy which prescribes a p as a function of q, that is we are looking for a function p(q) (sometimes this is called "reaction curve", particularly in Industrial Organization in Economics). To find the best response of player 1 we simply differentiate his expected payoff with respect to p. In this case we get $$ q - 2(1-q) = 3q-2 $$ This tells us that if player 2 plays q>2/3 then it is optimal to play p=1 as the expected payoff is strictly increasing in p. If q<2/3 then the opposite is true; the payoff is strictly decreasing in p and therefore p=0 is optimal. Finally, if q=2/3 then the derivative is 0 and the payoff is constant in q, so that any $p \in [0,1]$ is optimal. Thus, we found the following best-response curve
$$ p^{\ast}(q) = \left\{ \begin{array}{ccc} 1 &if &q>2/3 \\ \in[0,1] & if & q=2/3 \\ 0 & if & q<1/3 \end{array} \right. $$
Now, following the same steps using the player 2's expected payoff function we get
$$ q^{\ast}(p) = \left\{ \begin{array}{ccc} 1 &if &p>1/3 \\ \in[0,1] & if & p=1/3 \\ 0 & if & p<1/3 \end{array} \right. $$
Any intersection of these two best-response curves in the (p,q)-space (i.e., with p on the horizontal axis and q on the vertical axis) is an equilibrium of the game. It is easy to see that (p=1,q=1), (p=0, q=0), and (p=1/3 and q=2/3) are equilibria of this game.
I should say that the approach I sketched above is an overkill for such questions, and simple games are solved the way callculus did it. But it seems to me this is what you were trying to do.
Remark: more formally, to solve games we construct a best-response correspondence and then the set of fixed-points of such best-response correspondence is the set of all equilibria of a given game. That's why fixed-point theorems play such an important role in Game Theory.