I believe that you are talking about a popular first order axiomatization of some fragment of the theory of natural numbers, known as Peano Arithmetic.
First order means that the language only allows quantification over elements of the structure, in this case the natural numbers, but not over subsets of the structure.
If you allow quantification over subsets, finitely many axioms suffice to axiomatize the natural numbers. The tricky part here is induction, which is axiomatized in second order logic, i.e., with quantification over subsets, by saying:
For all sets $A$, if $A$ contains $0$ and is closed under taking successors, then $A$ is the set of all natural numbers.
It is impossible to formalize this is first order logic.
However, there is an approximation of this axiom: Essentially you say that every definable set of natural numbers has the property that if it contains $0$ and is closed under successors, then it consists of all natural numbers.
This can however not be expressed by a single axiom and requires an axiom schema consisting of infinitely many axioms. Namely, you need an extra axiom for each definition of a definable set.
Note that writing down an axiom schema usually requires a recursive definition of formulas. Since you want to define the natural numbers in order to be able to do recursion (for example) you might be unhappy with this.
But we need to base our development of formal mathematics on something, and in this case we base it on the fact that we know how to manipulate finite strings. I hope this clarifies things.