It is easy to say that physics would dissappear or be transformed, but those answers are vague and just speculative. So, here's an example how it can be approached. To start, concepts must be precise. So:
Mathematics has two essential functions: it is a language (allows communicating ideas), and it is a tool (allows performing calculations and getting new conclusions).
As a language, it is part of formal languages: languages that are defined in terms of concepts and axioms. Concepts are essentially ideas, and axioms are rules that are applied to concepts and preexisting objective ideas (e.g. Mathematics don't need to define what an object is). Check Kurt Goedel's theorem, is a good example on how concepts and axioms are used as the basis of mathematical formalisms.
As a tool, mathematics provides methods to perform mathematical calculus. Thus, it produces new rules (not anymore axioms, when they are consequences) and even new concepts (e.g. imaginary numbers). Again, Goedel's approach proposes a method of numbering all resulting consequential rules, essentially the result of mathematical calculus.
Physics is a scientific (axioms are empirical truths obtained following the scientific method) discipline (specified by a formal language) of knowledge which studies matter.
Now, the analysis is straightforward.
A) The linguistic dimension of physics is mathematics. So, if mathematics would be removed, physics would just lack of a mathematical representation.
What would left is still knowledge of the physical world, although expressed in other language: either a spoken language or another (e.g. Fortran? tally marks?). So, the second law of thermodynamics would be known, but could simply be expressed narratively (e.g. a cold body would never make a hot object hotter, if they enter in contact). This example uses an informal language (informal for physical purposes), but alternative formal languages exist. See Wikipedia for History of Mathematical Notation.
Consider that physical facts do contribute to mathematics, as the bra-ket notation, which raised from a need of representation of physical facts.
B) The tooling-related dimension of physics, physical calculus, is mostly performed by mathematics. But this as well, can be performed by alternative means. Mathematics turns to be just the most accessible option. But a lot of physical calculations can be performed mentally, without mathematics or by means of physical methods. Farmers don't use mathematics, but tend to optimize the exploitation of physical resources.
Finally, It should be noticed that you assume that math is part of physics, but such idea is just conventional (it depends on a strong boundary between what is physical from what is ideal). In fact, strictly, mathematics and physics study ideal objects, and both are inspired from empirical facts. Physics is said to study matter and energy, but both physics and mathematics are just abstractions of experience. The difference between ideal and physical is considered deprecated in philosophy, modern philosophy tends to accept that all entities are ideal, subjective. In such case, math would not be part of physics, but the opposite: physics would be part of mathematics, the part that is most related with sensibility, while mathematics would be the portion most related to rationality.