You ask:
Is math (only) a language?
Yes and no, depending which definition of language you are talking about, so let's discuss.
According to modern linguistics after contributions by Noam Chomsky borrowing from Saussure, language can be distinguished in terms of linguistic competence and performance. When most people use the term language, they are referring to performance which is the concrete act and product of language. If someone utters a sentence in Germany to buy a Trabbi, they are living up to an expectation of what a language is, mainly the use of sound to convey meaning to achieve pragmatic ends. But the notion of linguistic competence differs because it emphasizes not so much the process of communication, but the psychological capacity of agent to use language. It is therefore a form of psychological and sociological description. From WP:
In linguistics, linguistic competence is the system of unconscious knowledge that one knows when they know a language. It is distinguished from linguistic performance, which includes all other factors that allow one to use one's language in practice.
Thus, a language is not just the observable behavior of applying a grammar to communication, but is also an ontological framework for understanding how knowledge functions not only within the performance of language proper, but adjacent to language. For instance, one of the primary products of cognition both historically and contemporaneously is the notion of ideas, concepts, and thoughts. Language-as-competence is therefore a way to communicate about human cognition as well as communication.
As an example, consider the difference among the concepts of utterance, proposition, and assertion or judgement. An utterance is phonological and physical in nature; it is the phonemes and the phonics, the concrete implementation of communication. A proposition, however, is the semantic contents, the assembly of morphemes to convey the concept. And an assertion is the assignment of truth conditions to a proposition. Thus, there is a difference between recording the sentence 'The polygon is closed' with a recorder, understanding 'The polygon is closed' when listening, and determining that 'The polygon is closed' is true because the polygon is, indeed, closed in some drawing. Those are important distinctions in mathematical logic, and philosophy in general precisely because they are cognitive notions.
Consider as an example the Chomskyan generative grammar as a description of human cognition applied to the domain of mathematics. One of the most important introductions to the model after transformational grammar was offered in Syntactic Structures was the introduction of the lexicon and complex symbols to represent vocabulary in a language. In essence, it allowed one talk about language in virtue of the semantics. For instance, "The triangle has three sides" can be understood not just in terms of articles, nouns, verbs, and adjectives, but also in terms of quantity, polygons, properties, and line segments. Thus, there is no clear dividing line between mathematical concepts and mathematical language.
And what applies to the definition of a triangle, that there is no strict conceptual dividing line between a syntactic and linguistic interpretation and a semantic and cognitive interpretation applies to the entire endeavor of mathematics. Mathematics in this light, as Brouwer advocated, is a function of our linguistic intuitions (SEP). Part of mathematical language is performance, the explicit and behavioral process of using language faculty to convey ideas within the language community, but part of mathematical language is competence, the implicit and mental process of using language faculty to formulate and introspect mathematical ideas privately.
Therefore, yes, mathematics is a domain-specific language of quantity, quality, relations, operations, directions, and so on in both the the context of performance (submitting a mathematical theory for publication, for instance) and in competence (finding axioms, constructing theorems, corollaries, and lemma, ensuring they fit within the paradigm of the broader theory, finding applications for theories in modeling or other applications, etc.) Thus, the key to answering this question is understanding that language is not just about the syntax of communication; language always encompasses the generation and refereeing of semantics which is often called conceptualization. When seeing language as a linguist sees it, both performance and competence (or langue and parole to use Saussure's terms), math is certainly only a language.