Firstly, I am not sure what your question that comes under the title "Question" is. Let me try my best at it:
- The "ID" subgroup you seem to be interested in is called the identity subgroup, which for an abstract group $G$ is the singleton set $I=\{e_G\}$ where $e_G$ is the identity element of $G$. It is routine to check that $I$ is actually a subgroup of $G$.
For example, in $S_5$, $Id_{S_5}=(1)(2)(3)(4)(5)$ is the identity permutation which fixes all the five symbols whose group of permutations in $S_5$.
Your approach is right, but you fail to observe that, if $a \mid b$, then $a \le b$. This will get you out of those unnecessary contradictions.
Let $f$ be a homomorphism from $S_5$ to $\Bbb Z_{12}$. Then, you have following restrictions on $f$:
- The first isomorphism theorem, together with Lagrange's theorem tells you that, $|S_5/\operatorname {Ker} f|~~ \mbox{$=$} ~~|\operatorname{Im} f| ~~~\mbox{divides}~~ |\Bbb Z_{12}|$
So, for knowing the cardinality of $\operatorname{Ker} f$, notice that you need to find all those $x$ such that $\dfrac{120}{x} \mid 12$ Firstly, this in particular means that, $\dfrac {120} x \le 12$. So, you have that, $x \ge 10$.
As you know that only normal subgroups in $S_5$ are $1=\{\operatorname{Id}_{S_5}\}, A_5$ and $S_5$, you are forced to conclude that $\operatorname{Ker} f \in \{A_5, S_5\}$.
So,...