Probably the best indication that we have of Pythagoras's philosophy is the philosophy of Plato. He identified the monad of the Pythagoras with the form of the Good. Also, according to Hippolytus, the Pythagorean's said the monad begat the dyad which begat the points, which begat the limes, which begat the numbers, which begat finiteness and so on. Plato's take on this is described in his Timeaus, which although not taking directly this route, does use the Pythagorean notion of geometry being the underlying physical-mathematical structure of the world. From this, we can see that the Pythagorean use of numbers to describe their philosophy, wasn't purely number-theoretic, but a form of metaphor. We see an echo of this in descriptions of certain religions as monotheisms (mono - one) and of others, like Zorastrianism, as dualistic (dual - two). Merely because a number has been used in their description does not mean that they are focused on numbers per se.
In fact, Aristotle in book 1 of his Metaphysics whilst discussing the Pythagoreans at 990a, wrote:
how are we to understand that numbers and modifications of numbers of all being and generation both in the beginning and now, and at the same time there is no other number than the number that the universe is composed of?
Aristotle, in his critique of Pythagorean philosophy, instead described it as the first mover. This was part of his more scientific approach to Pythagorean philosophy. In this way he was illuminating the aporia in Pythagorean philosophy, as he saw it.
Simply because Aristotle critiqued Pythagorean philosophy does not mean that he rejected it. Critique is neccessary to improve one's understanding as well as the understanding of others. After, Einstein critiqued Newton, which does not mean he rejected Newtonian physics - he was setting out to improve it. Likewise with Plato and Aristotle with regards to Pythagorean philosophy.