Leibniz and Descartes are said to put forth "mechanist philosophies," but I am having trouble identifying what "mechanist" means. Does it involve their affinity to natural science and mathematics and their focusing on providing a philosophical foundation and response to the problems that arise in them? And does it include an extension of natural science and mathematics for metaphysical/ontological theories/explanations/arguments?
Mechanist (or mechanical) philosophy, in the original sense, meant the rejection of "substantial forms", i.e. forms with causal powers, such as souls, postulated by scholastics (who drew on some vague passages from Aristotle's De Anima). For a detailed discussion of substantial forms see How can the soul be a form in Aristotle's metaphysics? From the modern perspective, substantial forms are pseudo-explanations that "explain" X by postulating a causal power to do X. Moliere famously mocked this sort of metaphysics in "opium puts you to sleep because it has within itself a sleep-inducing power". As Leibniz described his struggles in 1661 (see SEP Leibniz’s Philosophy of Physics):
"After having finished the trivial schools, I fell upon the moderns, and I recall walking in a grove on the outskirts of Leipzig called the Rosental, at the age of fifteen, and deliberating whether to preserve substantial forms or not. Mechanism finally prevailed and led me to apply myself to mathematics."
In contrast, mechanistic explanations were supposed to explain everything by presenting it as a mechanism, a machine, operating according to spelled out (ideally, mathematical) rules, ultimately reducing everything to "particles in motion", in Descartes's quip. Boyle even used atomism as a litmus test for being a "mechanical philosopher". Of course, there is a difference between early rationalists and later pure mechanists (anticipated by Hobbes), who took Newtonian clockwork, complete with hypotheses non fingo, to its logical conclusion, with comparisons like brain produces thought the same way liver produces bile. Even Descartes had his "ghost in the machine", in Ryle's memorable label. And Leibniz's later monadology can hardly be called "mechanist" by any stretch, he even rejected atomism underwriting the particles in motion metaphor (monads were something of a spirtualized atoms).
At the same time, his philosophy of physics remained mathematically centered, and his idealism did not revert to substantial forms. Neither did Cartesian substance dualism, and Spinoza's Ethics, of course, took Euclid's Elements as its model of exposition (even Hobbes, who was more of an empiricist than rationalist, was enamored with Euclid). So, in this sense at least, new way of explaining things and mathematical reasoning did extend beyond the natural philosophy. But it is hard to identify any rationalist "program" beyond the natural philosophy. Leibniz, of course, attempted to mathematize everything and dreamed of calculus ratiocinator, a conceptual calculation framework that was supposed to resolve all metaphysical and epistemological problems. But that was very undeveloped, much of it unpublished, and hardly widely shared.
In the 18th century, the two "mechanist" ideologies played out in a number of controversies, such as the vis viva controversy and the dispute about the nature of space, and the general competition between the more qualitative and "romantic" Cartesian natural philosophy, and the drier, more "mechanical" and mathematical Newtonian physics. By the end of 18th century, Newton's more precise and mathematical approach prevailed, and the mantle of natural philosophy was picked up by German romantics, increasingly hostile to modern science, mathematization and mechanical explanations, see Was there early opposition to Newton's mechanics?