There is an area of specialization within many mathematics programs focussed on 'Foundations', or 'Symbolic Logic', (although that latter is often 'Symbolic Logic and Combinatorics', and may lie closer to hard-core computer science than philosophy in some programs.)
If you find a relatively 'old school' advisor, courses in the history and philosophy of mathematics (and/or science) are still part of this 'Logic' concentration, and will be sometimes cross-listed with philosophy and also with computer science, as some old, deep ideas still influence 'Cognitive Science' and AI. Interdisciplinary studies at that 'triple-point' are sometimes encouraged.
Taking this path as broadly as possible does not prepare one to 'pursue philosophy professionally', just to have a more meaningful, humanistic basis behind your mathematics. At the same time, a terminal Master's in Mathematics is also not generally adequate to constitute 'pursuing mathematics professionally'. Instead, it provides a more abstract and more thoroughly grounded basis for some kind of computing, scientific, or engineering career, or for teaching those who will pursue such careers.
There is also no shame in multiple Bachelor's degrees instead of a Master's or multiple Master's degrees instead of a Doctorate. All forms of thinking reinforce one another. And given the ubiquity of education and the way jobs have changed, breadth and flexibility may trump dedication in a market where people don't work "in their field" most of the time.
And if it is a real issue, you might want to work this out at the MS level. From personal experience, I do not suggest attempting a Doctorate in math if your loyalties are in any way divided. The level of focus necessary to even 'qualify as a candidate' much less find an advisor and a problem, in a good program does not leave room for indecision.