First of all, the latter requirement in your Question is too strong: clearly you want to be able to infer $P \to Q$ from $P \to Q$ and $\neg P$. And saying that $\phi$ cannot be inferred unless it is equivalent to either $P \to Q$ or $\neg P$ doesn't work either: we still want to be able to infer $\neg P \lor R$.
Still, while your requirement as stated is too strong, we are indeed left wondering: how exactly are we to understand these 'things in particular' that we should not be able to infer from $P \to Q$ and $\neg P$?
Well, I think it is pretty clear that we should not be able to infer either $Q$ or $\neg Q$. That is, the following argument should be invalid:
$P \to Q$
$\neg P$
$\therefore Q$
And to do that, we need a counterexample with $P = F$ and $Q = F$, and thus we should set $F \to F = T$
Likewise the following argument should be invalid:
$P \to Q$
$\neg P$
$\therefore \neg Q$
and that means that we have to set $F \to T = T$.
OK, but what about $T \to T$? Would setting it to $F$ mean that suddenly the truth of $P \to Q$ does enable us to infer 'things in particular' from the falsity of P? Well, clearly not!
Consider: Suppose we set the three truth-values as already established. Then whatever statement $\phi$ we can infer from $P \to Q$ and $\neg P$ with $T \to T = T$, we can of course also infer from $P \to Q$ and $\neg P$ with $T \to T = F$, because any counterexample to any inference will require $P$ to be false, and for that case, the two possible definitions work exactly the same, meaning they have the exact same set of counterexamples, and thus have the exact same set of sentences that are logical consequences.
Thus: it is not the case that not being able to infer 'things in particular' (however they might be defined) from $P\ to Q$ and $\neg $ will force the truth-table for the material conditional to be what they are!
OK, so is there an independent argument for setting the truth-values as we do? Here is one:
$T \rightarrow F = F$. OK, this makes immediate sense intuitively, but if you want a more technical argument, you can indeed just point to Modus Ponens:
$$P \rightarrow Q$$
$$P$$
$$\therefore Q$$
Now suppose $P = T$ and $Q = F$. If $T \rightarrow F$ were $T$, then this argument would be invalid! Clearly that's not what we want. So, we should set $T \rightarrow F = F$
Now let's consider:
$$P \rightarrow P$$
OK, clearly we want this to be a tautology, no matter what $P$ is saying, and no matter whether $P$ is true or false ( Indeed, even if $P$ is a contradiction, it should still hold that ' If P then P'!). OK, but this means that we can't set $T \rightarrow T$ to $F$, for then $P \rightarrow P$ would not be a tautology, so we set $T \rightarrow T = T$. Likewise, we can't set $F \rightarrow F$ to $F$, so we set $F \rightarrow F = T$.
Finally, we want $\rightarrow$ to be 'asymmetrical' or non-commutative: clearly 'if P then Q' is completely different from 'if Q then P'. But given the other truth-values already set as they are, if we set $F \rightarrow T$ to $F$, then it would become commutative! So, we set $F \rightarrow T =T$.
In short, setting the truth-values as we do is the only way to ensure:
Modus Ponens is valid
$P \rightarrow P$ is a tautology
$\to$ is non-cummutative
And, just to have some more arguments for setting the truth-values as we do, consider:
$$P \rightarrow Q$$
$$Q$$
$$\therefore P$$
This should clearly be an invalid argument, with the counterexample of $P = F$ and $Q = T$. But if we were to set $F \rightarrow T$ to $F$, this would not be a counterexample at all! So, we better set $F \rightarrow T = T$.
Finally, let's note that we want:
$$P \rightarrow Q \Leftrightarrow \neg Q \rightarrow \neg P$$
This means that $T \rightarrow T$ and $F \rightarrow F$ better have the same truth-value. So, once you are convinced that one of them should be $Qt$, then this contraposition equivalence should convince you that the other should be $T$ as well.