The short answer is that pure mathematics has to be free from empirical determinations of any kind (not just physical, but also psychological) to be what it is. In other words, knowledge becomes essentially mathematical when it can be considered neutral in its relationship with experience of any kind.
Three observations arise from this preliminary point:
1) Mathematicians themselves live in the world of experience (and so will any kind of concrete mind that thinks mathematically), and that is why this desirable "purity" of any mathematical object is, in practice, a process, not something that can have a referent in the world. That can be said, regardless of metaphysical considerations - for instance, if mathematical objects exist or not (which is a big philosophical problem, check this out: https://youtu.be/1EGDCh75SpQ).
2) The connections between mathematical knowledge and its "applications" in natural science is not trivial, at all. Your could even say that, whenever they are spotted for the first time, these connections normally appear surprising, uncanny, even overwhelming, for both sides. Physical science can be about nature, but it is not itself "natural". When science changes the way we see the world, it changes the world, and itself. So my answer to your nuclear question is that the patterns that mathematicians formulate are thinkable, and the patterns that empirical scientists identify are evident. Whatever happens between these two stances is game, and the game of science is not made of trees and branches, it is whatever scientists do. It may very well be that one day knowledge will no longer be so easily departmentalized. Many authors have discussed this trend, and I won't try to be encyclopedic here, so let me just suggest this reading (Isabelle Stengers): http://we.vub.ac.be/aphy/sites/default/files/stengers2011_pleaslowscience.pdf.
3) In the end, the utility of knowledge is in the eye of the beholder.