Short Answer
'What is mathematical' is one of the primary concerns of the philosophy of math, and depending on one's beliefs about and in philosophy, particularly one's metaphysics, there are very divergent views on what it means to be mathematical. An approach to defining 'mathematical' (besides as a synonym for 'precise') might fall under two approaches, one a general, natural language definition that might conform to prototype theory and one that follows more technical questions of intension that draws on questions of necessity and sufficiency.
A general response to 'mathematical' would follow the common-sense definition that would invoke terms such as quantity, qualities, relations, shapes, operations, truth, and direction and would appeal to some common systems of mathematics like arithmetics, algebras, vectors, and geometries. The technical response to what is mathematics and where it comes from would be more in line with the various schools in the philosophy of mathematics. These responses are far more technical but include ideas like reduction to logic, formal systems, intuition, empirical, abstraction, iteration, and semantics. The former class of characterization is comprehensible to most people, whereas the latter class is technical philosophy drawing particularly on the jargon of epistemology and ontology.
If what is 'mathematical' were voted upon by the set of educated adults, the former class would be the most popular, but whether or not one finds that appealing would be related to one's views on the use of ordinary language in philosophy and the sophistication of one's philosophical jargon. It should be said, that any claims to certainty about what is 'mathematical' would conflict with any epistemological approach that is pluralisitic and fallibilistic (IEP). The claim that Pythagoreans threw competitors off a cliff (Reddit) is likely to be apocryphal, but it certainly is believable.
Long Answer
The Politics of Mathematics
Any answer to what is mathematical is a lot like an intellectual fingerprint. Take for instance the mathematical philosophy of Wittgenstein, it differs remarkably from the claims about 'mathematical' made by Aristotle (accounting for the fact that mathema, 'μάθημα' in Greek languages is something much broader than what we call mathematics).
To demonstrate how the definition of 'mathematical' is highly partisan and divergent, let us consider the characterization of the 'mathematical' from an empirical perspective and what the SEP has to say:
he general philosophical and scientific outlook in the nineteenth century tended toward the empirical: platonistic aspects of rationalistic theories of mathematics were rapidly losing support. Especially the once highly praised faculty of rational intuition of ideas was regarded with suspicion. Thus it became a challenge to formulate a philosophical theory of mathematics that was free of platonistic elements. In the first decades of the twentieth century, three non-platonistic accounts of mathematics were developed: logicism, formalism, and intuitionism. There emerged in the beginning of the twentieth century also a fourth program: predicativism. Due to contingent historical circumstances, its true potential was not brought out until the 1960s. However it deserves a place beside the three traditional schools that are discussed in most standard contemporary introductions to philosophy of mathematics, such as (Shapiro 2000) and (Linnebo 2017).
If one takes the position that mathematical thought (whatever it may be) reduces to psychological theory such as recognized since the psychologism of the 19th century, then one might align with John Stuart Mill's mathematical philosophy (SEP) as an inductivist endeavor:
Amongst the Laws of Nature learnt by way of inductive reasoning are the laws of geometry and arithmetic. It is worth emphasizing that in no case does Mill think that the ultimately inductive nature of the sciences—whether physical, mathematical, or social—precludes the deductive organization and practice of the science (Ryan 1987: 3–20). Manifestly, we do work through many inferences in deductive terms—and this is nowhere clearer than in the case of mathematics. Mill’s claim is simply that any premise or non-verbal inference can only be as strong as the inductive justification that supports it.
This approach to understanding 'mathematical' may not be as popular as Husserl's, it rejects transcendentalist notions derived from Kant that an eliminative materialist would be okay with. Linnebo in his Philosophy of Mathematics has a chapter on contemporary empirical interpretations of 'mathematical'.
Empirical, Intuitional, and Unorthodox Accounts
Linnebo raises the a point about the relationship between the empiricism of Locke and mathematical thinking on page 88:
[T]he very idea that mathematical knowledge is synthetic a priori clashes with the empiricist creed that all substantive knowledge is empirical.
He goes on to cover Mill and Quine's contributions.
In Chapter 8: Mathematical Intuitionism, he suggests that intuitionalism is empirical in nature but rejects the excesses of empiricism on page 116:
We shall now take a closer look at some accounts of mathematical knowledge that do not assimilate it to empirical knowledge... [since] some form of mathematical intuition provides evidence for certain mathematical truth.
One unorthodox approach to characterizing mathematics comes from George Lakoff and Rafael E. Núñez in their Where Mathematics Comes From. Broadly speaking, the answer to the title of the book is psychological processes of the mind that can be understood as conceptual metaphors. One example they give in their book is the relationship between the minds fundamental linguistic capacity to categorize and using the Metaphor of Containment as understanding the intuitive sets of naive set theory. It's an obvious empirical truth that people understand containers long before they develop formal mechanisms for defintions of extension. Of course, while the idea that the brain is strongly related to the mind through neural correlations of consciousness, there is still a lot of woo in mathematical philosophy (thus revealing my bias).
Summary and Advice
If you're looking to understand what is 'mathematical' and what is not, like most fields in including attempts to define 'science', there can be friction over demarcation and definition. If you're looking to understand what is 'mathematical' and you don't have some experience in undergraduate math, it might help if you learn a little non-Euclidian geometry, abstract algebra, and formal set theory before wading into the very abstract and technical notions of mathematical philosophy which has a very fascinating explosion of ideas since Frege Gottlob invented the basis for the formalisms used today in mathematical logic. At the bare minimum, as you develop your own beliefs, you'll be able to understand some of the basic theories that underlie technical notions of mathematics and begin to see as Thomas Kuhn did of science, that mathematics is conducted by mathematicians, many of whom get paid to defend their beliefs or have professional reputations (not to mention feelings) at stake when they peddle philosophies that may largely reject contemporary science and persuade because of the practical outcomes of staying on the bandwagon.