It is easy to say that physics would dissappear or be transformed, but those answers are vague and just speculative. So, here's an example how it can be approached. 

To start, concepts must be precise. So:

Mathematics has two essential functions: it is a language (allows communicating ideas), and it is a tool (allows performing calculations and getting new conclusions).

As a language, it is part of _formal languages_: languages that are defined in terms of concepts and axioms. Concepts are essentially ideas, and axioms are rules that are applied to concepts and preexisting objective ideas (e.g. Mathematics don't need to define what an object is). Check Kurt Goedel's theorem, is a good example on how concepts and axioms are used as the basis of mathematical formalisms. 

As a Tool, mathematics provides methods to perform _mathematical calculus_. Thus, it produces new rules (not anymore axioms, when they are consequences) and even new concepts (e.g. imaginary numbers). Again, Goedel's approach proposes a method of numbering all resulting consequential rules, essentially the result of mathematical calculus.

Physics is a scientific (axioms are empirical truths obtained following the scientific method) discipline (specified by a formal language) of knowledge which studies matter. 

Now, the analysis is straightforward. 

A) The linguistic dimension of physics is mathematics. So, if mathematics would be removed, _physics would just lack of a mathematical representation_. 

What would left is still knowledge of the physical world, although expressed in other language: either a spoken language or another (e.g. Fortran? tally marks?). So, the second law of thermodynamics would be known, but could simply be expressed narratively (e.g. a cold body would never make a hot object hotter, if they enter in contact). This example uses an informal language (informal for physical purposes), but alternative formal languages exist. See Wikipedia for History of Mathematical Notation.

Consider that physical facts do contribute to mathematics, as the bra-ket notation, which raised from a need of representation of physical facts. 

B) The tooling-related dimension of physics, _physical calculus_, is mostly performed by mathematics. But this as well, can be performed by alternative means. Mathematics turns to be just the most accessible option. But a lot of physical calculations can be performed mentally, without mathematics or by means of physical methods. Farmers don't use mathematics, but tend to optimize the exploitation of physical resources. 

So, why does this strong correlation between mathematics and physics exist? This is a speculation of my own: Essentially because of this: mathematics deal with ideal objects and physics... too! A free fall experiment will never produce the exact result that mathematics predict (having one decimal of tolerance is accepting that the expected value is different from the predicted value; but the right result would have all possible infinitesimal digits identical in the prediction and the measurement). We assume a lot of ideals. So, physics would be just some form of mathematics inspired by empirical observations.