I'm trying to figure out why exactly the argument from tolerance fails. What I've come to so far is that to practice relativism means to see a cultures morals from a standpoint that of a person within the culture, while tolerance is understanding that cultures morals differ, and not holding subjective opinions against it. What would cause this to fail?
I parse the question into this: why cultural relativism (CR) cannot be justified from the standpoint of tolerance? For this, let's first clarify what CR is. CR can be descriptive or prescriptive. As a descriptive idea, CR states the observation that morality is relative to each culture. As a prescriptive, moral theory, CR asserts that morality must be founded on each culture. All moral values are nothing more than customs, social norms and/or legal practices. I focus on this moral theory side of CR in this post.
As a (prescriptive) moral theory, CR is already very problematic, CR as a moral theory is based on the so-called naturalistic fallacy (inferring what one ought to do from what one is). Also, under CR, a moral revolutionist (like M.L. King) becomes a common criminal. CR cannot explain why stoning couples for marrying across different castes is wrong. Indeed, CR can defend any clearly morally wrong practice insofar as the practice is the norm in the society. Consequently, CR cannot offer guidance for moral progress. For this reason, no moral theorists are motivated to defend CR.
Setting aside the fact that CR is an inadequate moral theory, can CR be defended nevertheless for the reason that it is based on the value of tolerance? Surely, tolerance has some goodness going on. Like individuals, each culture has its own idiosyncrasies due to existential contingencies. Many cultures around the world find it offending when some globally dominant culture judges their cultural norms as morally wrong or inferior. Not exercising tolerance can be viewed as imperialistic or paternalistic.
Does this means that CR can be justified from the standpoint of tolerance? If tolerance is an intrinsic value, then CR can be, but it is not. Tolerance can be good or bad. While we should be tolerant towards differences in general, there are times when being tolerant borders on acting cowardly: e.g. tolerating a bully. Tolerance is valuable when it serves some higher values. John S. Mill, for instance, argued for tolerance for the sake of liberty, which he viewed as a source of human happiness (cf. David Brink's Mill's Progressive Principles). John Rawls argued for tolerance (through epistemic modesty and neutrality on the good) to show the ideal of equal respect. Since tolerance itself has no steady foundation, CR justified by tolerance will be even more unstable, which is why the argument from tolerance to CR fails.
There is an issue of scale. This is a perfectly good political value. As an ethics, it is beyond human capacity.
Tolerance is a reasonable thing for a legal system to have. For people, it is an obnoxious thing. It readily falls apart into two different things: 1) acceptance, or 2) passive aggression.
We cannot 'tolerate' an alternative view without knowing it is wrong and therefore considering it lesser, which undercuts our ability to truly deal with the holder as an equal. So what we are really doing is passive aggressively refusing to act against something we actually actively disagree with. We can agree to disagree right up until it matters, at which point we have to either accept or deny it, in order to move forward. Meanwhile, that puts us in a false relationship that requires a lack of authenticity in order to avoid the issue.
There is no argument that a legal system anchored in some larger cultural value cannot be tolerant, up to the brink of logical consistency. But true cultural relativism between individuals is not psychologically tenable. We can legitimately accept their right to live out their principles, or we cannot. Tolerating them until they clash with ours is just evasion.