The fundamental question is: Does the order of my choices matter?
In this problem it does not: it’s the same five players whether you select them in the order A, B, C, D, E, the order A, C, D, E, B, or any of the other $118$ possible order. Your calculation, however, counts the selection A, B, C, D, E as different from the selection A, C, D, E, B. In other words, you’ve counted each permutation of $5$ players as a separate selection, when all that is wanted is the number of distinct sets of $5$ players that can be formed.
If, on the other hand, the problem had been to count the number of ways of selecting $5$ of the digits $0,1,\dots,9$ and writing them down to form a $5$-digit number (possibly with a leading $0$), the order would have mattered: the number $24815$ is different from the number $84512$, even though they contain the same digits. Your calculation would have been correct for this problem.
Your second question is about the multiplication (or Chinese menu) principle of counting. You have two choices to make: you must select the members from the math department, and you must select the members from the computer science department. The numbers here are moderately large, so lets look at a similar problem with smaller numbers. Suppose that the committee is to have just two members, one from each department. Let’s also suppose that the math department has only $3$ members and the computer science department only $4$. Make a little table:
$\begin{array}{r|cc} &C_1&C_2&C_3&C_4\\ \hline M_1&1&2&3&4\\ M_2&5&6&7&8\\ M_3&9&10&11&12 \end{array}$
Each number in the body of the table represents one way of picking a committee: number $7$ for instance, is the committee $\{M_2,C_3\}$ consisting of mathematician number $2$ and computer scientist number $3$. Clearly there are $12$ possible committees, simply because there are $12$ slots in the table; and there are $12$ slots in the table because it has $3$ rows and $4$ columns, and $3\cdot4=12$.
Now imagine making a similar table for the bigger problem. It would have $\binom93=\frac{9!}{3!6!}$ rows, one for each of the possible sets of $3$ mathematicians, and it would have $\binom{11}4=\frac{11!}{4!7!}$ columns, one for each of the possible sets of $4$ computer scientists. It would therefore have $\binom93\binom{11}4$ slots, each of which would correspond to matching a particular set of $3$ mathematicians with a particular set of $4$ computer scientists.
The general principle is that if you are making a set of $m$ independent choices, and choice number $k$ can be made in $n_k$ different ways, then the whole task can be carried out in $n_1n_2\dots n_m$ different ways. In my simplified committee problem $m=2$, $n_1=3$, and $n_2=4$; in the original committee problem $m=2$, $n_1=\binom93$, and $n_2=\binom{11}4$.