Sure, those philosophers exist. But they're not nearly as trendy as you might have imagined. The philosophical arguments made against affirmative action now are exactly the same ones that were made by theorists long before race-conscious theories ever existed. It's the writings of anyone who ever encouraged blind justice and universal social equality, from strict deontologists like Immanuel Kant to liberal political philosophers like John Rawls.
One could do worse than to start by thinking through Rawls's Veil of Ignorance, wherein one imagines that she is forced to decide the fate and organization of her future society without knowing the specific role in that society which she will be assigned to play. In other words, no one knows their place in society, their class position, their social status, or their assets and abilities ahead of time. They might be placed into the society as a poor black female, a rich white male, or anything in between. The idea is that one would choose the most just society for all members, rather than be tainted by a bias for one's own current social group. This is nothing particularly revolutionary, considering that liberal theorists since the Enlightenment have argued in favor of universal equality, justice as fairness to all, and the equal (re-)distribution of wealth.
In fact, it is the notion of "affirmative action" that is relatively new, and not just because philosophers have stereotypically been rich white men that didn't care about the oppressed minority groups. Philosophically and legally, the affirmative action system has its roots in critical legal studies (CLS), and more specifically, the branch of CLS that deals with race—critical race theory (CRT). The primary authors here are probably Derrick Bell and Richard Delgado, and I think Mari Matsuda deserves specific mention as a particularly lucid and readable theorist in this tradition. Her book (with Charles Lawrence), We Won't Go Back: Making the Case for Affirmative Action, is probably one of the landmark texts tying CRT with affirmative action.
Critical race theory has two primary objectives. First is a critical genealogy of the way in which power struggles across racial boundaries have been borne out over time, and more specifically, the role that the law and legal system have played in that process. Second is a sustained effort to transform the current system (both legally and socially) into one that is capable of true racial emancipation and a sustainable ability to resist entrenched oppressive hierarchies. The Wikipedia article is a fine introduction to the other themes at work in CRT (see the section entitled "Key theoretical elements), but the real landmark of CRT, the pivotal point of disagreement between it and traditional liberalism as discussed above, is the rejection of color-blindness. In summary, CRT is a race-conscious theory that looks specifically at the dynamics of racial power and their effects on the law, rather than pretending that law is racially-neutral as liberalism has done for years. CRT theorists have argued that abstract, rights-based remedies don't work, as they fail to change the oppressive hierarchies of subordination and dominance that have been for so long entrenched into the legal system. As hard as we pretend that the system is color-blind, the results indicate clearly that it is not. (There's a lot of other critical theory hoopla mixed in, of course, like intersectionality, strategic essentialism, and a strong emphasis on personal narratives, but none of this is particularly germane to the issue that is addressed in the question.)
Essentially, though, it all boils down to how one feels that equality should be measured. There are two primary camps here. The first argues that equality should be assessed in terms of one's opportunities; this is commonly referred to as the equality of opportunity. The argument here is that everyone should be equally capable of achieving the same ends, that there should be no systemic barriers (social, legal, cultural, economic, or otherwise) that stand in the way of the success of any individual or group. The other primary theory of equality is commonly referred to as the equality of outcomes (or results), in which the argument is made that true equality is only achieved when everyone in the society ends up in truly equal positions after all the decisions have been made and everything has shaken out. This gives prejudices and discrimination a chance to rear their ugly head, intervening in what appeared on the surface to be a completely fair and equitable system. Which type of equality that you consider most important to maximize generally determines which types of social programs you favor.
