Your urge seems to be the otherworldly allure of old-fashioned Platonism, and the very reverse of most modern tendencies, which is nice. In the ancient sense, Geometry indicates a form of completion, purification, or "self-enclosure" that at last eliminates the contingencies and irrationalities of physical experience and presumably transcends its own origins.
This idea, which was last resurrected by Frege and Russell, seems to be in general disfavor now, in part for good ideological and political reasons. By reducing mathematics to set theory and logic, the hope was to make it a self-completing, purely "analytical" system, ultimately a tautological construction grounded in axioms, if not the old faulty Euclidean axioms. (For Russell, one effect was actually to inoculate "reasoning" in the real, natural world from its proclivities towards metaphysical "totalitarian" prescriptions. Analytics can model the world through science, but cannot ultimately "prove" anything necessary about it.)
But if mathematics is to be purely axiomatic, what are the axioms to be founded on? Intuition? Clear and distinct ideas? And how are we to arrive at universal intuitions? The problem is an ancient one that became more acute when it was determined by Gauss, et al. that Euclid's system was not fully deducible from his axioms, and that relatively consistent, perfectly useful systems could be generated from other axioms... though never, it turned out, mutually and completely reduced to one axiomatic system. Of course, Godel is said to have delivered the coup de grace to this hope of completion.
As others have pointed out, physics might take this not as a defeat but as a triumph, for it retains the relative certainty of mathematics, while allowing its uncanny, seemingly unlimited involvement in the physical world. We now have more attempts to derive math from physics, the near inversion of the effort to reduce it to logic. Penelope Maddy, for one, attempts a set theory realism, not unlike your coffee cup process, which might then be correlated to mathematized "neural assemblies."
One interesting "in between" theory would be Kant's once notorious classification of mathematics as "synthetic a priori" as opposed to purely analytical. In other words, it is a priori idealism, not based in physical experience, yet neither is it a tautological system of analytical deductions. Like induction in physics, it produced or "synthesizes" new knowledge... knowledge that is not absolutely "certain" and potentially complete, but is "as certain" as rational processes can get for human consciousness, given its necessary, categorical structure.
There are, of course, many developments of mathematics that are highly speculative, far from any physical origins, and seem to have no physical applications, a possible example being Cantor's infinities or even Godel's platonism. Can the mathematician then follow these increasingly pure forms and wean himself from their physical origins? Cantor went mad, "left his senses," and Godel starved himself to death. So perhaps there is a "way out" for you, if that's what you really want.