The answer is straightforward in the context of Chomsky's universal grammar, which music does not fit. However, the innate grammar structures postulated by Chomsky were not as universally encountered outside of the major European and Oriental languages, and the conception has little purchase with modern linguists, see Does majority of linguists accept universal grammar? So let us compare music to conventional languages outside of that context.
Leibniz would seem to agree with the OP, he called music “an unconscious exercise in arithmetic in which the mind does not know it is counting”. But despite the well-known phenomenon of "lost in translation", most English texts can be "adequately" translated into German, Chinese, Swahili, and vice versa. Many pages though it would take, but even texts of specialized languages with terms like "decoherence" or "symplectomorphism", can still be translated into colloquial English. But in an English "translation" of say Vivaldi's Four Seasons almost everything that matters will be lost, and that even for an uncharacteristically "contentful" piece.
First thought is to distinguish music from conventional languages by saying that it lacks the content or meaning, but few musicians would concur that music is meaningless. Still, it looks like there is an equivocation going on here, "meaning" is used with two different meanings, pardon the pun. When Frege was developing the first modern semantics he distinguished two different components of "meaning" (actually three, but the third is moot here), sense and tone. In more recent times Frege's theory of tone was developed by Dummett, see Kortum's Varieties of Tone. The sense is that side of meaning that determines the "propositional content", the truth conditions of a sentence, what would make it true. It is therefore also essential to determining its reference, if any. Tone, which Frege also called "coloration" or "illumination", has little effect on those:
"It makes no difference to the thought whether I use the word ‘horse’ or
‘steed’ or ‘nag’ or ‘prad’... What is called mood, atmosphere, illumination
in a poem, what is portrayed by intonation and rhythm, does not belong to
the thought".
Schopenhauer, who developed an influential philosophy of music, expressed himself more poetically, "music is by no means like the other arts, namely a copy of the Ideas, but a copy of the Will itself". It is not that music is entirely contentless, but it strongly prioritizes tone over content, whereas natural languages prioritize content over tone. Their expressive means are lined up accordingly, hence the futility of translation. "Rules" for stringing notes into musical phrases are more aesthetic/psychological heuristics than structural prescriptions of conventional grammar. One could of course construct a soritic chain like logic - rhetoric - poetry - chanting - singing - music, and "dissolve" the Fregean distinction with the no-boundaries-no-difference trick so popular in postmodernism. But if music is a "language" it is not just another language, it is a language of another sort.
"Music is the universal language of mankind" is credited to Longfellow, although apparently the expression was used before him, "though music be a universal language, it is spoken with all sorts of accents", Shaw added later. The metaphor of "language" is often used informally because while music largely does not convey content (and lacks reference), it is generally felt that it does convey something. If the point of languages, especially theoretical languages, is primarily to inform and refer, the point of music is perhaps to evoke, emotions, moods, attitudes, volitions, etc. Apparently, that is also the point of Frege's tone. So we could say metaphorically that music is a language of qualia.
I'll end with Schopenhauer:
"The inexpressible depth of music, so easy to understand and yet so inexplicable, is due to the fact that it reproduces all the emotions of our innermost being, but entirely without reality and remote from its pain… Music expresses only the quintessence of life and its events, never these themselves."