Good question; it relates to the notion of judgement, which for Kant is part of aesthetics; which shows that aesthetics, when philosophically thought, isn't the conventional notion - that of art; it is of course related in some sense.
Though you've asked about how the right definition is chosen in mathematics, similar questions can asked elsewhere; for example, why does a novelist choose one word over another? or an architect one material over another? the choice of artists and artisanry is deliberate; because where craftsmanship is exercised, judgements must be made; and these judgements become encoded in the tradition of that art or craft; and in this sense, overtly objective disciplines, such as physics and mathematics, as well as being a science, are also arts; that is crafts.
This is Grothendiecks famous description of his approach to mathematics:
The unknown thing to be known appeared to me as a stretch of earth, or hard marl, resisting penetration...the sea advances insensibly, in silence; nothing seems to happen, nothing moves; the water is so far off you hardly hear anything; yet it finally penetrates the resistant substance.
And of course, mathematicians talk about good mathematics, or beautiful mathematics or what ought to be the case (the Kantian triple, in a sense; but mostly not) and this, to some extent, justifies the description above.
It is one of the purposes of a good mathematical education to enstill these values - usually called mathematical maturity - which are in a sense, unencodable, into the human substance; hence the institutions and traditions of this craft.
To come back to your specific question; higher-dimensional geometry was developed post the advent of coordinates by Descartes, (that 'honest' thinker in Kierkegaards phrase, who used doubt as a tool and not as a scalpel - critique rather than criticism); in that language it's easy to generalise that most basic of identities - the one of Pythagoras; no visualisation is necessary as would have been in the geometric imagination; one can then push through calculus in this context using the technology of vector spaces and then manifolds; in both cases, there is a natural metric, a natural way of measuring distance, which is related at a basic level to the Pythagorean theorem; in a different direction, metric spaces began to be investigated; these are just pure spaces with a different but related notion of distance, one even more basic than Pythagoras - I mean the triangle identity; it's hard to argue with, since it seems so natural; and so basic; and thus one might argue this ought to be enough; but it was noticed in many arguments that the metric wasn't necessary and one could use open sets; this is an important innovation, because it drops the measuring rod and instead, for lack of a better word, uses 'place', the word used by Aristotle in his investigation of space; in a way it measures space by space itself - the measurement is commensurable (like changes like, so like measures like).
Aristotle, in his physics, linked this notion, Place, to ideas of change and variation, contiguity and continuity, and of the parts to the whole - all important notions.
It's important to link the development of these notions to others; topology as a discipline wouldn't have been conceivable; or rather, it could be conceived as a Concept, and it was done so; but it couldn't be determined as an Idea (to steal a term from Hegel); that is placed in sharp relief in the axiomatic manner without a certain amount of technology; and that technology is what is now called Set Theory; and this Notion carries the basic idea that things are made of elements, of points. This is the modern view; the post-modern view is that extensionless points aren't neccessary, or are wrong, which was the view argued by Aristotle, in his investigation of continua; one can in fact, drop the points of a topological space, and just investigate its cohesive structure - it's topology - thus locale theory, or in a phrase that should be more famous than it is, 'pointless topology'.
One can go further, homotopy theory is a sub-discipline in topology; and this uses the notion of paths; after much development, and many results; it was noticed that the subject could be 'axiomatised', and this is called abstract homotopy theory, and one of its main tools is 'model categories'; then in an another innovation, it was noticed that this was a 'coordinatisation' of infinity-categories: for not only are there paths between points, there are paths between paths; this again is an ancient observation, that can be traced to Parmenides, 'doesn't each notion of likeness, give rise to another, and then again'.
Still, this shows that open sets are only one step, though a good strong step, in the long tradition of investigating space and continuity.
Finally it's worth pointing out, that historically speaking that open sets were not the only way, then of tackling continuity; given that limits by sequences are the basic tool in analysis, one might expect it would be this tool that would be generalised; and this is possible - they are called nets, and not widely used; they have a dual form - filters - and these are used more widely; but open sets introduces a new idea, or rather return to us an old one, that space is cohesive, and that it can be 'unglued' into small pieces - note, that open sets have extension, that they are not points, and that their extension is not measured by a number - and then 'glued' back together; and it is this notion of 'gluing' which generalises to a more general context like sheaves and stacks.
One might do without the 'Judgement of God' (and it sometimes appears that in this world one must); but one cannot do without Judgement altogether: culture, in its widest sense, being essential to man, in part his essence objectified, is also the objective manifestation of judgements; also too, in its widest sense.
In Little Gidding, a poem by TS Eliot, he writes:
We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time
and this is the repeated motion of the mathematical mind meditating on space; broadening and deepening it; by repetition and difference; but each time returning to the beginning and seeing the same ground, again; the same earth, again; as though for the first time, afresh; and thus -
To become renewed, transfigured in another pattern