The key is to give the definition of homology in a module chain by the language of category, and then generalize it to the Abelian category.
Let A→B→C be a module chain with f:A→B, g:B→C, gf=0. Then the homolgy at B is just the cokernel of the embedding i:im f→Ker g. So we want to define i by the language of category. Note that f=jf1, where f1:A→im f has the same definition with that of f on A, j:im f→B is the embedding. gf=0 implies gjf1=0, then gj=0 by f1 being epi. So there is a unique morphism h: im f→Ker g so that j=kh, where k is the kernal mapping k: Ker g→B.
We then show that h is just i. This is due to direct computation. Given b in im f, b=j(b)=(kh)(b)=k(h(b))=h(b). The fisrt equality holds because j is embedding, similar arguments to the last equality.
Now we want to know whether the way to define h(=i) is a way of using the language of category. We fisrt decompose f into the composition of a epimorphism followed by a monomorphism, which is not justified in an arbitrary category, but justified in an Abelian category (this is what Abelian does!). Furthermore, the range of the epimorphism in an Abelian category is just the image of f(this can be proved, but not very easy), which concides with the modular case. Then we construct h just by the universal property of the kernel of g. So we can define h in an Abelian category. But we now cannot ask whether h=i, because i makes no sense in an arbitrary category. But we have already shown that h=i in the modular case, so we can use h directly instead of i in the arbitrary case, which will give a generalization(this is why we needn't know, we just want a generalization!).
So the definition follows, it's just cokernel of h: im f→Ker g, where h is defined above.