Suppose that $N$ students decide independently of each other to enroll in (i) class A only, (ii) class B only, (iii) both class A and class B, and (iv) neither class A nor class B, with probabilities $p$, $q$, $r$, and $s$ respectively where $p+q+r+s=1$. It is observed that $n_1-n_{12}$ students
are enrolled in class A only,
$n_2-n_{12}$ students are enrolled in class B only, and $n_{12}$ students are in both classes. As noted by Sasha, we have a multinomial distribution, and the likelihood of this observation is thus
$$q(N; n_1, n_2, n_{12})
= \frac{N!p^{n_1-n_{12}}q^{n_2-n_{12}}r^{n_{12}}s^{N-n_1-n_2 + n_{12}}}{(n_1-n_{12})!(n_2-n_{12})!n_{12}!(N-n_1-n_2 + n_{12})!}.$$
To find the value of $N$ that maximizes $q(N; n_1, n_2, n_{12})$, we look at
the ratio
$$\frac{q(N; n_1, n_2, n_{12})}{q(N-1; n_1, n_2, n_{12})}
= \frac{Ns}{N-n_1-n_2+n_{12}}$$
and note that the ratio is greater than $1$, (that is,
$q(N; n_1, n_2, n_{12}) > q(N-1; n_1, n_2, n_{12})$), if
$$N < \frac{n_1+n_2-n_{12}}{1-s} = \frac{n_1+n_2-n_{12}}{p+q+r},$$
and smaller than $1$, (that is,
$q(N; n_1, n_2, n_{12}) < q(N-1; n_1, n_2, n_{12})$), if the above
inequality is reversed. In other words, the maximum-likelihood
estimate of $N$ is
$$\hat{N} = \frac{n_1+n_2-n_{12}}{p+q+r}
= \frac{\text{total enrollment in classes A and B}}{P(\text{student enrolls in A or B or both})}$$
where floors and ceilings have been ignored for simplicity of exposition.
Of course, if we do not know $p+q+r$, this estimate is not very useful.
Varying $p+q+r$ to find the maximum value of $\hat{N}$ is futile. It leads to
an estimate of an infinite number of students none of whom take courses A or B. So much for higher education!