(a) $P(X
Suppose $q = P(3 \text{ before } 5)$. When will we see a $3$ before we see a $5$? That is if we first see a $3$ on the first roll (we are done), or if we don't see a $3$ or a $5$ on the first roll and then see a $3$ before a $5$ after that. In other words:
$q = P(3) \cdot 1 + P(5) \cdot 0 + P(\text{neither } 3 \text{ nor } 5) \cdot q$
Since we know that $P(3) = \frac{2}{36}$, $P(5) = \frac{4}{36}$ we get $P(\text{neither } 3 \text{ nor } 5) = 1 - \frac{2}{36} - \frac{4}{36} = \frac{30}{36}$, so
$q = \frac{2}{36} + \frac{30}{36} q \quad \Longleftrightarrow \quad q = \frac{1}{3}$
Note that we can also rewrite the first equation as
$q = P(3) \cdot 1 + P(5) \cdot 0 + P(\text{neither } 3 \text{ nor } 5) \cdot q \quad \Longleftrightarrow \quad q = \frac{P(3)}{P(3) + P(5)} = \frac{P(3)}{P(3 \text{ or } 5)} = P(3\ |\ 3 \text{ or } 5)$
This is in accordance with a comment made by Dilip: We can simply ignore all throws which are neither a $3$ or a $5$, as they are irrelevant. Then the probability of seeing a $3$ first is simply the conditional probability of throwing a $3$, given that it's either a $3$ or a $5$.
(b) $P(X
We can use the same approach as above. First, let us ignore all throws which are not $3$, $5$ or $10$. We now need that the first event is a $3$, which happens with probability
$P(X
Now if this happens, we only need that in all successive events of $3,5,10$ we first see some number of $3$s, and then see a $5$ (and not a $10$ yet). But that means that after this first event, we can again ignore all events $X$, and look at the first throw resulting in either a $5$ or a $10$. Then
$P(Y
So the combined probability is then
$P(X