In case you haven't already figured it out, I like to ask questions about how to measure soundness.

Question a

Suppose I haven two arguments, X and Y, each consisting of p premises. Suppose that argument X commits f > 0 fallacies, whereas argument Y commits g fallacies, where 0 < g < f. Could one say that argument X is worse than argument Y in terms of its soundness?

Question b

Suppose there are two entities, X and Y, each having made a arguments that average p premises each and have the same distribution of the number of premises per argument. Suppose that entity X commits f > 0 fallacies in total, whereas entity Y commits g fallacies in total, where 0 < g < f fallacies. Could one say that entity X is worse at making sound arguments than entity Y?

Question c

In Question a, what if the two arguments have a different ratio of fallacies to premises?

Question d

In Question b, what if the ratio of fallacies to arguments differs between the two entities?

Question e

In Question b, what if the distribution of the ratio of fallacies to premises differs between the two entities?

  • I think it's fairly obvious that fewer fallacies is better, no? Can you give one reason why more fallacies could possibly be better? As it stands, I don't see a real question here. – Ben Aug 14 '13 at 14:51
  • @Ben The alternative I'm interested in isn't that more fallacies is better. The alternative is that the number of fallacies doesn't matter. Only the presence or absence of any fallacy does. So, yes, there is indeed a "real" question here as opposed to, I don't know, a fake one. – Brash Equilibrium Aug 14 '13 at 15:24

I like to ask questions about how to measure soundness.

Unfortunately, before we can go forward, we need to unpack some of the assumptions lying behind your question.

To begin with: measure the soundness of what?

Arguments of some kind, of course-- but are these formal arguments, or natural language (i.e., rhetorical) arguments?

If they are formal arguments, as soon as there is a logical fallacy, the argument fails. Additional fallacies don't make much difference; it is like a mathematical proof with a erroneous step.

If, however, you are speaking of informal arguments, i.e., arguments in natural language, then the fallacies that you are speaking of are rhetorical in nature, and don't actually invalidate the argument at all. They fail to support the argument, but don't undercut it from a logical perspective (although they may make it less persuasive psychologically.)

Note that this very question demonstrates the problem that we are dealing with.

Much depends upon how we interpret the terms "argument" and "fallacy"-- whether we are speaking of them in the context of formal logic, or in the context of rhetoric. This ambiguity is a hallmark of natural language arguments, and necessarily so: words have meaning in context, and the context is never completely saturated.

In other words, attempts to apply the tools of mathematics (ratios of fallacies to premises) are bound to fail, as there is no way to unambiguously count premises or fallacies, since their recognition depends upon an interpretive process dependent upon context, and that context (in an non-formalized structure) cannot be sufficient for epistemic closure.

  • Excellent answer. I am dealing with rhetorical arguments that I ultimately wish to "score" in terms of their adherence to facts and logic. I agree with everything you said, but add that although I can't reach epistemic closure, I can estimate some underlying, latent structure to the argument and its validity by asking a number of people to reconstruct in a pseudo formal way and tag it appropriately with fallacies committed. Then I could somehow incorporate the uncertainty in the structure of the argument and the identified fallacies, likely using Bayesian inference. – Brash Equilibrium Aug 15 '13 at 13:50
  • 2
    You have two options, it seems to me. One is to just admit that you are being subjective, and assign scores to the arguments based on the strength that you perceive. The other is to attempt to formalize the arguments, but this is a famously impossible task-- in fact, just trying to deduce whether or not a give chunk of text contains an argument, or to identify the premises within the text, seems to be far beyond current AI research. – Michael Dorfman Aug 16 '13 at 9:39
  • We aren't using AI to start. We are using multiple raters. – Brash Equilibrium Aug 16 '13 at 14:57
  • Then you're back to option one: to recognize that you are working with subjective judgment, and that your scores will be subjective measures and not objective. – Michael Dorfman Aug 18 '13 at 11:03
  • 2
    Perhaps; but that's more a function of the qualities of the people involved (grading or upvoting/downvoting, etc.) – Michael Dorfman Aug 19 '13 at 15:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.