The question itself has been already discussed by the previous answers, but I think that understanding the history and the purpose of mathematics and how it differs from other sciences would be of great help in general.
As someone studying mathematical logic/meta-mathematics. I can assure you that the purpose of mathematics is not to describe the universe, and that mathematics does not describe it. Physics does, chemistry does, biology does, natural sciences in general do — but mathematics certainly doesn't.
We consider that both writing and mathematics (through counting) were created around 8000 BC as a way to keep trace of agricultural goods. Nowadays mathematics are a way to express, understand and share complex ideas in general.
During antiquity there was no such thing as logically constructed mathematics (that does not mean that mathematics were not constructed that way but rather that they were not consciously constructed that way). For example, if someone argued that 1 + 1 does not equal 2, he would have been ignored for being stupid and not understanding 1+1=2. But no proof of 1+1=2 would have been given. Any proof would just have been a speech in natural language that would have convinced the majority. As a consequence, most of the work done until Renaissance was not more advanced than what can be done with an abacus and ruler & compass maths.
Around the 18th/19th century mathematicians realized that both mathematics and natural languages were too complex, that proceeding that way lead to errors and misunderstandings. They worked on a way to guarantee that a proof was correct and they could understand each other properly.
At that point mathematical notations and logic were created (Leibniz and Peano played a major role in this and despite being hard to read by today's standards I would still recommend to everyone to read their books/papers).
Notations allows people to universally understand exactly what we are talking about, and logic is a rather simple tool that allows us to make proper proofs.
The base of formal mathematics is axioms : a set of rules that we take for granted. Axioms cannot be proven, they have to be extremely carefully chosen. As long as we assemble, mix the axioms following a set of rules, the result is guaranteed to be logical, error-less and the whole proof is a coherent thought.
But if the axioms are not chosen properly we can prove nonsense. For example if the axioms used consider two opposite propositions to be true, then everything can be proven according to those axioms.
In maths we arbitrarily define rules basic enough to build upon in a safe (logical) way. But nothing prevents you from creating a custom set of axioms, and build upon them to create a description of the opposite of our universe.
In physics (and other natural/experimental sciences) we chose a set of axioms basic enough not to allow us to make errors and try to define an equation describing some natural phenomenon. Then we do experiments trying to show that the equation we built was wrong, if we fail such a high amount of times that we can consider the equation to be sufficiently accurate and reliable we use it.
Maths is a powerful tool to describe complex things in general. But it is just a way to describe cognitive abstractions in a standardized format.
Also mathematics have a limit. Gödel's incompleteness theorem states that no matter how full and complex the set of axioms we chose, there will always be truths that we will not be able to prove.