Here's a bit of a tricky way to do it: instead of maximizing $x^2 y^3$, we'll maximize its logarithm $2 \log x + 3 \log y$, or alternatively $2 \log x + 3 \log (3 - 3x/4)$.
But that denominator of $4$ is annoying; since $ \log (3 - 3x/4) = \log (12-3x) - \log 4 $ we can minimize $ f(x) = 2 \log x + 3 \log (12-3x) $ instead.
Differentiate: $f^\prime(x) = 2/x - 9/(12-3x)$. This is zero when $2(12-3x) = 9x$, or $24-6x = 9x$, or $x = 8/5$. Since $3x+4y=12$ we can solve to get $y = 9/5$. The maximum value of $x^2 y^3$ is therefore
$ (8/5)^2 (9/5)^3 = (4/3 \times 6/5)^2 (3/2 \times 6/5)^3 = (4/3)^2 (3/2)^3 (6/5)^5 = 6 (6/5)^5. $
This is probably not the most efficient solution to this particular problem. (This is made worse by the fact that once you get the answer, it still takes a while to figure out which of the choices it is; I suspect it's not what the writers of whatever test you're preparing for had in mind.) But I give it because it illustrates a general principle: products are annoying, so if you want to maximize one sometimes it's easier to maximize its log, which is a sum.