I'm a philosophy professor at a research university, with a PhD in philosophy. My experience, if you couldn't guess, is that it depends. For philosophy of math, physics, logic, and, yes, even language (as it's practiced today), you need to know some (but not too much) recondite math. You also need to be proficient with it. That is, having taken the courses won't suffice. In the case of philosophy of math, for instance, you really need to understand Compactness, Lowenheim-Skolem, Incompleteness, axiomatic set theory, Inner Models and Forcing, even if you don't contribute to logic yourself. You also want to be familiar with non-classical foundational frameworks, like Quine's NF or type theory, and non-classical logics, including modal, intuitionistic, and paraconsistent logics. Finally, you should have familiarity with formal theories of truth, especially Tarski's and Kripke's, and will want to know something of 'mainstream' mathematical programs, like Langlands.
Nat is right that if you do 'old-fashioned' epistemology or philosophy of mind, you won't need to know much more than formal logic. But a lot of contemporary philosophy, even historical, assumes competence with technical machinery. The situation is really not analogous to Art History or English. This is witnessed by the many interdisciplinary math/philosophy and related PhD programs, like Berkeley's Logic & Methodology of Science (founded by Tarski), Carnegie Mellon's department, Notre Dame's joint philosophy/mathematics PhD, Pittsburgh's HPS, Maryland's Philosophy of Physics program, etc. It is also witnessed by the many scholars in philosophy departments with PhDs in math, physics, computer science, or a cognate field, like Tim Bays (Notre Dame), Hugh Woodin (Harvard), Scott Weinstein (UPenn), Harold Hodes (Cornell), Haim Gaifman (Columbia), Michael Potter (Cambridge), Steve Awodey (Carnegie Mellon), John Burgess (Princeton), David Wallace (Pitt), David Albert (Columbia), Joel Hamkins (Notre Dame), Shaughan Lavine (Arizona), and so on.
Philosophy is a strange subject: it encompasses so much. It's not very useful, then, to look at GRE scores for "Philosophy" PhDs. That will lump together students at Carnegie Mellon with students at New School for Social Research. "Philosophy" in each department a distinctive thing. Even within a single department, some will ally with humanities, others with sciences. The best way to gauge what math you need is to read the philosophy that interests you. If you want to understand, e.g., Kit Fine's The Limits of Abstraction, a work of metaphysics, you're going to have to know a fair bit of logic and be able to follow long, technical proofs. But that's not the case with, say, Bernard William's Ethics and the Limits of Philosophy. Both are brilliant philosophical contributions, by the way, and plenty of students can mimic the first, but not the second, or vice versa.