First of all I'm not an expert in this field, please correct me if I'm lacking relevant knowledge here. A few hundreds years ago mathematics was largery based on intuition. People realised we need to make mathematics more rigorous. Axioms and definitions came into play and people have chosen to derive new theorems from those axioms and definitions by using propositional logic. But the axioms and definitions were formulated by mathematicians to reflect some informal ideas, such as a definite integral, or some much more abstract like those in topology. How are mathematical definitons formulated? We start with a specific example, find some more examples and notice patterns, features these examples have in common. Let's take definite integrals as an example. It's a result of our attempts to find the area under the graph of a function. What if I formulated the Riemann integral to be the limit of sum of infinitely many infinitely thin rectangles having height f(x)? Today we know that the existence of limit is not enough here, we need to make sure the choice of sample points doesn't affect the value of limit.
So basically I don't know if a new definition is correct until I find a counterexample. A definition has to be simple, capture my intuitive idea of the area under the curve, but not too narrow (I have to impose certain restrictions, i.e. the function has to be integrable) and not too broad (I don't want to define a definite integral so that in special cases it cannot be considered as the area under the curve).
If you look at topology in mathematics, it's full of some crazy definitions that seem to have come from nowhere. Indeed, it took mathematicians a few decades in 20th century before the majority accepted this set of definitions. If it took so long to formulate them, then the earlier definitions had to be "worse" in some sense. Does it mean the current ones are perfect? For some reason, probably not. Maybe it was more like: here is our (mathematicians of 20th century) idea of what topology should be all about, and finally here's the set of definitions that fullfills these needs. But we cannot be sure it does - we can find a counterexample showing that those definitions are not "complete" in some way, leading to serious paradoxes etc.
So my question is - how to make sure a definition captures the intutive reasoning correctly, taking all special cases into account? I'm guessing the answer is: you can't do that, because definitions are axioms. But remember, we cannot do maths without the use of intuition, no matter how rigorous we are trying to be. Even Russell's Principia Mathematica assumes the reader will interpret the magical symbols in the same way as the author. The assumption that we are all thinking in the same way "on a certain level" - let's allows us to do mathematics. Imagine a small child learning how to speak - does he know any language he can refer to in order to learn how to speak? Obviously not. We learned human speech as kids from specific examples of how and when certain sentences and words are used, in which situations. We can communicate with each other because the way we interpreted these examples was identical on some level.
The problem I'm trying to emphasise here is that we actually can develop theorems from certain definitions (axioms) for years and then realise the axioms were wrong in some way, incorrect with our intentions. Then it'd be practically impossible to "repair" this theory. Mathematics is correct until we find a flaw in it.
One more, important example:
Continuous function. In the times of Newton and Leibniz, mathematics was largery based on intuition. They developed a theory without being precise in what limit or continuity meant, they didn't have a formal definition of these notions. The intuitve motivation for the definition of a continuous function is the ability to draw a graph of it without lifting the pencil from a plane. This isn't rigorous. The aim was to find a formal definition. It was first formulated as: function f is continuous at point x if:
No matter how hard I try, I can't find an example that this definition doesn't agree with this intuitive notion of continuity. Still, there is a chance someone smarter will find it. Why do we confirm such a definition by giving examples showing that it works? We can never prove anything by examples (we can only disprove a theorem by providing a counterexample)! Somehow all mathematicians agree this definition is correct. What makes them think so?