I like Geoffrey Thomas's answer, and would add.
Mathematicians and rationalists more generally have always chased after certainty of reason. Plato was perhaps the first mathematical foundationalist who attempted to deal with infinite regress by creating a distinct, objective reality for abstraction with his theory of forms. For over two thousand years, Euclid's postulates and axiomatic method in Elements were seen as an exemplar in reasoning and imitated, since axiomatic reasoning relies on foundations. Rationalism with Descartes through works like Meditations sought to establish metaphysical certainty of rationalism. With mathematicians like Frege who came along and advocated a logicist position, ultimately the epistemological foundations of math were debated regarding the acceptance of non-Euclidean geometries which had to deal with how to be certain one has distinct, correct postulates. This was a major philosophical issue in the 19th century. While accepting that there were not necessarily right and wrong axioms, mathematicians also had to deal with Cantor's infinite paradise and the pursuit of set-theoretic axiomatic foundations of set theory which at the time were hotly contested. Men like Cantor bumped heads with Kronecker. These debates eventually lead to Hilbert's goal of certainty crushed by Kurt Gödel in the 20th century.
But the axe you are specifically referring to is a question regarding the role of intuition in mathematical reasoning. There is a school of thought called intuitionalism which sharply differs with that of formalism. A classic example of this is how geometric intuitions and analytical foundations are used in calculus. What is a curve? Since calculus, the derivative, which is a measure of slope of a line tangent to a curve at a point has generated a discussion about what constitutes a curve. Remnant of Zeno's paradoxes, the question of what a differential is invokes foundational questions in the same way that Cantor's transfinite numbers do. These maths work and were accepted, but the nature of their foundation either through intuitive concepts like 'parallel lines' or 'continuous curve' creates issues in reasoning axiomatically. Non-Euclidian geometries open up a bag of worms in terms of psychologism because if one can intuit contradictory, but functional postulates, how can one be certain about one's foundations at all? Characterizing mathematics as essentially a type of logic, would for mathematicians, kick the can down the road by asserting all mathematical foundations are certain, because they reduce to logical statements (leaving open the question of where the logical statements are ultimately grounded).
While non-Euclidian geometries open up the question, a reduction of math to logic would make things certain. This works to an extent. Consider the adoption of ZFC in the early twentieth-century in response to the ambiguities of intuition in naive set theory. ZFC postulates are commonly given in terms of quantitative and existential qualifiers. The null set axiom can be stated as "There exists some set such that for all elements pertinent to that set, there does not exist a relation such that any element exists in that set, and can be denoted by a special symbol called the empty-set symbol". (∅≡∃S∀e¬(e∈S)) Now, a mathematician can say, since the logic is accepted, the mathematical axiom is beyond doubt.
This is the axe. Some people see statements which rely on intuition as pathological (a metaphor from disease). And because they distrust intuition, they rally against formalisms built on it.
What I am interested in is what were the philosophical axes they had to grind?
As far as the philosophy of math can be extricated from philosophy in general (and I would argue it would be difficult to do well), I would suggest that there are several at play which are all interlinked:
intuitionism v. formalism (mathematical) which reduces to
psychologism v. objective realism (19th century philosophical) which reduces to
empiricism v. rationalism (18th century, philosophical) which reduces to
what is the appropriate level of skepticism, the relationship between the mental and physical, and tackling ideas like the Agrippan trilemma which purports to show that rationality ultimately fails to provide truth or certainty.
Remember that most of the 19th century occurred before Wundt, James, and Freud established the legitimacy of psychology as a science. So psychologism v. objective realism is really linked to the credibility of what we now called psychology which hadn't been established. In the 21st century, cognitive science is pushing back on objective realism by arguing exactly how neurons create concepts, and establishing the mind-body duality as a category mistake. But back then, mind-body duality was widely accepted as a philosophical truth. To this day, many philosophers outside of the philosophy of mind are vigorously defensive of the notion of objectivity in contradistinction of intersubjectivity, and the debate continues.