In short: The so-called definition of natural numbers as those that can be obtained from 0 by adding 1 repeatedly is circular, but there is no viable alternative, which already makes it impossible to uniquely pin down natural numbers mathematically. Worse still, there does not seem to be ontological reason for believing in the existence of a perfect physical representation of any collection that satisfies PA under a suitable interpretation.
Why the so-called definition is circular
It is circular because "repeatedly" cannot be defined without essentially knowing natural numbers. You cannot use the natural numbers to do counting because you have not defined them yet! You are stuck; you must already know what are natural numbers before you can talk about iteration. This is why in mathematical logic the meta-system must already have the collection of natural numbers to be able to define what it means for a formal system to be arithmetically sound (prove only arithmetical sentences that are true in the 'true natural numbers'). And for a slightly less brief account of various assumptions needed in the foundations of mathematics, see this post.
Why there is no viable alternative
The problem comes right at the beginning even before you can talk about arithmetic. Think about just pure propositional logic. What is a well-formed formula in propositional logic? To even define that, you necessarily must assume the existence of finite strings in the real world, otherwise you cannot even have a precise syntactic form for sentences, not to even say a full logic system. Furthermore, it is not enough to have a constructivistic view of finite strings in the sense that you can recognize them while not necessarily generating them. This is because propositional logic has deduction rules that permit deducing "A ⇒ A ∨ A" for any propositional formula "A", which clearly implies that one must be able to generate arbitrarily long strings. Effectively, to even describe the deductive system for propositional logic one already must accept the existence of the collection of finite strings.
One might naively think that perhaps we do not need formal systems at all. But the only way we can precisely and objectively describe something to another person is by a finite sequence of symbols in a common language. Pictures do not work because they are subject to interpretation unless they are in an agreed fixed format, in which case they could easily be encoded by symbol strings anyway. And if we want to have logical reasoning, the mere notion of mathematical proof involves finite sequences of symbols, hence by accepting any formal system whatsoever as being meaningful, we already must accept the basic properties of string manipulation, which amount to accepting TC (the theory of concatenation). But TC (despite having just the concatenation operation and no arithmetic operations) is essentially incomplete, so we cannot pin down even the finite strings!
So we do not even have hope of giving to anyone a description that uniquely identifies a collection of finite strings, which naturally precludes doing the same for natural numbers. This fact holds under very weak assumptions, such as those required to prove Godel's incompleteness theorems. If one rejects those... Well one reason to reject them is the following...
Why there is no apparent physical model of PA
As far as we know in modern physics, one cannot store finite strings in any physical medium with high fidelity beyond a certain length, for which I can safely give an upper bound of 2^10000 bits. This is not only because the observable universe is finite, but also because a physical storage device with extremely large capacity (on the order of the size of the observable universe) will degrade faster than you can use it.
So description aside, we do not have any reason to even believe that finite strings have actual physical representation in the real world. This problem cannot be escaped by using conceptual strings, such as iterations of some particular process, because we have no basis to assume the existence of a process that can be iterated indefinitely, pretty much due to the finiteness of the observable universe, again.
Therefore we are stuck with the physical inability to even generate all finite strings, or to generate all natural numbers in a physical representation, even if we define them using circular natural-language definitions!
Further facts
There are two curious facts related to this. Firstly, despite the fact that PA (Peano arithmetic) is based on the assumption of an infinite collection of natural numbers, which as explained above cannot have a perfect physical representation, PA still generates theorems that seem to be true at least at human scales. My favourite example is HTTPS, whose decryption process relies crucially on the correctness of Fermat's little theorem applied to natural numbers with length on the order of thousands of bits. So there is some truth in PA at human scales.
This may even suggest one way to escape the incompleteness theorems, because they only apply to deterministic formal systems that roughly speaking have certain unbounded closure properties (see this paper about self-verifying theories for sharp results regarding the incompleteness phenomenon). Perhaps the physical 'fuzziness' due to quantum mechanics or the spacetime limitations may permit the real world to be governed by some kind of system that does is syntactically complete, but anyway such systems will not have arithmetic in full as we know it!
Secondly, any meta-system MS that can reason about finite strings and sets of finite strings can prove the incompleteness theorems about itself, which immediately implies that if MS is consistent then MS' = MS + ¬ Con(MS) is consistent but Σ1-unsound (from the perspective of MS). But think about it: How do you know that MS is not already Σ1-unsound? The unsoundness could be buried deeper as well. MS'' = MS + ¬ ω-Con(MS) is ω-consistent but of course not arithmetically sound.
The thing is that we don't have any way to decide whether or not our preferred meta-system (whether ZFC or something nice and predicative like ACA) is in fact arithmetically sound, until we actually find a proof of say "⬜..⬜(0=1)" for some number (possibly zero) of "⬜"s. We cannot just say that if nobody has found such a proof then it is good enough evidence that they do not exist, since Godel's speed-up theorem and elegant examples by Harvey Friedman show that it is possible for the shortest such proof to be so long as to be impossible for humans to discover by trial and error.
