Why do we say that the Peano Axioms define the natural numbers when there are more models other than natural numbers for those axioms? Maybe it is a confusion relative to the word "define", which I understand as a description that, when applied, unambiguously refers to the definiendum's referent.
It is only the first-order Peano theory that has non-standard models. The second-order Peano theory has only one model up to isomorphism.
A second-order theory of arithmetic makes use of at least one second-order axiom such as the Axiom of Induction which says that you can do induction on natural numbers or the Axiom of Well-ordering which says that the less-than relation is a well-ordering (every set contains a least element). The original Peano arithmetic is second order because it uses the Axiom of Induction.
The first order Peano theory is generally defined by using the original Peano axioms except for Induction (and some axioms now treated as part of the logic), and adding a few more, including a countable set of axioms to replace the Axiom of Induction. First order Peano arithmetic has non-standard models.
Why would you use a first order theory instead of second order when the first order theory has non-standard models? Because of completeness. First order logic is "complete" in the sense that any sentence true in all models is derivable from the axioms. Second order logic is not complete in this sense, so you have to chose between two non-ideal situations: a theory that is complete but has non-standard models or a theory that is not complete but has only standard models.
There are two strategies that I am aware of.
First, note that compactness in FOL gives us nonstandard models. See here: https://mathoverflow.net/questions/40821/existence-of-an-omega-nonstandard-model-of-zfc-from-compactness. So we need to find a logic where compactness does not hold (and where a couple other nice properties hold). It turns out that this is second order logic.
Second, Tennebaum 1959 showed that there was no countable nonstandard model in which we can compute with the nonstandard numbers (roughly). So we can use this as a characterization also.
I don't think "the Peano Axioms define the natural numbers" is a sentence that mathematicians say. But we can fix it and recover a lot of value. Instead, let's say "the Peano Axioms are true for the natural numbers."
That's an assertion, not really a theorem (what base assumptions would you take in order to prove it???) but I think most everybody believes it. Additionally, the basic rules of first-order logic are pretty uncontroversial.
So we have a couple conclusions we can draw:
- We believe that PA is true on "the" natural numbers (whatever those are); and
- We believe that first order logic "works;" and thus
- We believe that any theorem provable from PA is also true for the natural numbers.
Since PA and first-order logic are pretty easy to work with, this gives us a way to prove statements about "the" natural numbers in a way that's mathematically convenient and coming from a baseline that basically everybody agrees with. Moreover, it turns out these axioms are powerful -- you can prove a lot of theorems just from PA. So it's a useful set of axioms.
That's not to say they're the only set of useful axioms. Some people work with additional axioms (specifically, the idea that the natural numbers are embedded in a field of real numbers, can be divided into sets, analyzed topologically, etc.) and can prove more theorems from that. These axioms are still pretty uncontroversial, but they're inconvenient in some ways because of the amount of "surrounding baggage" they bring in.
Some people take even stronger additional axioms around category theory and large cardinals and so on -- you don't need to know what those are, but you should know that they are sometimes controversial, and not everyone thinks they are true, but they do allow you to prove even more theorems.
The point is we have options about what axioms to take. But PA is a shared baseline that is small enough that almost everybody agrees with them, but big enough that we can prove lots of stuff from them. So they're useful.
Hope that helps!