I think that they can be viewed like that, with some suitable definition of philosophy.
Then mathematics could be defined as one of the branches of philosophy in which theories are built on definitions and axioms and the results are proven and physics can be thought of as some kind of philosophical theory of laws of nature (you know the full Latin name of Newton´s book Principia) that are seeked both experimentally and by constructing mathematical models.
I know this is a naive question, and it reveals my amateurism in the field, but, does this makes any sense to you?
I do not see anything particularly unphilosophical in math and physics, so, what would be some problems if we would define them to be branches of philosophy?
Would then some change be needed in the definition of the scope, range and reach of philosophy?