Short Answer
In philosophy, there are questions that have pat answers, such as what was Kant's primary language, and then there are questions that do nothing but raise more questions. 'What is mathematics' is of the latter. The philosophy of mathematics addresses this/these questions, and provides as a starting ground for additional reflection. There is the general definition of 'what mathematics is' according to MW:
Definition of mathematics
1 : the science of numbers and their operations (see operation sense 5), interrelations, combinations, generalizations, and abstractions and of space (see space entry 1 sense 7) configurations and their structure, measurement, transformations, and generalizations Algebra, arithmetic, calculus, geometry, and trigonometry are branches of mathematics.
2 : a branch of, operation in, or use of mathematics the mathematics of physical chemistry
But philosophical definition might be better be understood as the art of characterization. Three common ways mathematics is understood are through logicism, intuitionism, and formalism. If you develop a basic understanding, you're on your path to having your own answer on 'what mathematics is'.
Long Answer
What is mathematics is a tremendously broad question by today's standards of philosophical discourse. But it is an absolutely recurring question for those who are starting to reflect on metaphysical principles. A coherent philosophical response can occur on different levels. If you have a background in category theory and are familiar with the history of Hilbert's program, it may take one form. I'm going to presume you're not a graduate student in mathematics and answer this as if you are just becoming interested in the philosophy of mathematics. Certainly, not everyone who is interested in the philosophy of math can calculate tensors and eigenvectors.
This broad question places you firmly in the domain of encyclopedia entries. Start with Philosophy of Mathematics (SEP).
A quick summary from this article suggests four contemporary schools:
2.1 Logicism
2.2 Intuitionism
2.3 Formalism
2.4 Predicativism
Let's take a look at the first three!
Logicism
Logicism is very popular and sprung from the work of Gottlob Frege who is some consider one of the fathers of the linguistic turn in philosophy. Although relatively unknown, Bertrand Russell brought attention to his work and championed it alongside some very famous philosophers such as Alfred North Whitehead. They may have written a work about it.
From the article:
the philosophy of mathematics, logicism is a programme comprising one or more of the theses that — for some coherent meaning of 'logic' — mathematics is an extension of logic, some or all of mathematics is reducible to logic, or some or all of mathematics may be modelled in logic.3 Bertrand Russell and Alfred North Whitehead championed this programme, initiated by Gottlob Frege and subsequently developed by Richard Dedekind and Giuseppe Peano.
For a taste of this philosophy of mathematics which is largely a reduction of mathematics to formal logic, consider the logicist conception of the number system and arithmetic encompassed by Peano's Axioms (PA), one of the elemental achievements in mathematical logic. From Wolfram:
- Zero is a number.
- If a is a number, the successor of a is a number.
- zero is not the successor of a number.
- Two numbers of which the successors are equal are themselves equal.
- (induction axiom.) If a set S of numbers contains zero and also the successor of every number in S, then every number is in S.
Doesn't seem very 'mathy', does it?
Intuitionism
Intuitionism is a school which rejects the notion that mathematical objects are somehow 'out there', and accepts that math is an activity of the human mind:
In the philosophy of mathematics, intuitionism, or neointuitionism (opposed to preintuitionism), is an approach where mathematics is considered to be purely the result of the constructive mental activity of humans rather than the discovery of fundamental principles claimed to exist in an objective reality. That is, logic and mathematics are not considered analytic activities wherein deep properties of objective reality are revealed and applied, but are instead considered the application of internally consistent methods used to realize more complex mental constructs, regardless of their possible independent existence in an objective reality.
Given our understanding of the working of the brain, and married with our knowledge about computers, it might be hard to see how philosophers might reject this, however, a longstanding understanding of mathematics was a product of Platonism which might be oversimplified as the belief that in some sense, circles are 'real' objects that exist in some alternative world of being that we have access to through our thoughts. Just as a modern-day Christian might believe in an actual place called Heaven populated by actual beings called Angels, a platonic thinker might believe there is a Real of Forms inhabited by real objects called Circles. As a naturalist who rejects supernaturalism, I find both ideas distasteful, but that doesn't mean my beliefs are true. These are fundamental discussion in metaphysics related to ontology and epistemology.
Formalism
If you are a computer scientist, formalism may be the approach for you! With its emphasis on defining mathematics as operations on formal systems, there is a certain appeal to seeing mathematics as visual language games with complex rules, like the lambda calculus.
(I'd cite some typical fare from a highly abstruse formal representation, but PhilSE doesn't do math formatting well.)
This approach emphasizes how mathematics can be shown to be computable. David Hilbert himself was a proponent of this aspect of mathematics given his love of formal systems.
What Is Mathematics and Where it Comes From
These are broad questions, but after working your way through some encyclopedic entries, you might want to start with some primary sources. A classic text is Bertrand Russell's Philosophy of Mathematics. There are contemporary authors like Oystein Linnebo's Philosophy of Mathematics which covers other lines of thought that intersect with mathematics such as nominalism. Ultimately, though, be prepared to learn some mathematics along the way, since the logic of mathematical discovery often feeds the logic of mathematical philosophy.
There is no simple answer to defining mathematics, and like any philosophy, questions tend to grow faster than answers. But if you ask enough questions, eventually you'll find some satisfaction in the collection of answers.