16

As a layperson, I try not to fall victim to the Dunning-Kruger effect. As an example, one area where I know that I am vulnerable is when biochemistry intersects with nutrition and disease.

Is it possible for me, as a layperson without specific scientific knowledge (in this case, biochemistry), to make any sound or valid judgment regarding a scientific dispute (in this case, over nutrition)? For instance, the China study says X, critics A say not-X, and critics B say Y. Isn't any attempt by a layperson to evaluate any of those claims going to be based on guesswork and confirmation bias?

  • 1
    I'm not sure there is a really satisfying answer for the general case -- it depends on your ability and willingness to acquire some degree of scientific rigor. I am curious what others might make of this. – Joseph Weissman Aug 25 '11 at 15:45
  • 1
    Appeal to authority is a fallacy of irrelevance only when the authority is irrelevant to the argument. The argumentative authority of a biochemist is not relevant in an argument about astrophysics, but is very relevant in an argument about biochemistry. – Mitch Aug 25 '11 at 17:17
  • 3
    I think you've successfully avoided the Dunning-Kruger effect at the point where you ask this question. More specifically, you recognize your incompetence when it comes to subject matter in which you are not knowledgeable, thus avoiding the "illusory superiority" that Kruger and Dunning warn about. You haven't miscalibrated your own cognitive or intellectual abilities at all. – Cody Gray Aug 26 '11 at 2:20
  • 2
    And that's really the important thing to take away. It's basically impossible for anyone to become an expert on a subject overnight, and certainly it isn't possible for one to become an expert on all subjects. The real goal of education is to expand your horizons such that you appreciate all of the things you don't know. – Cody Gray Aug 26 '11 at 2:21
  • 1
    Something I would add: If, as a layperson, you pick a side in a scientific dispute, you are engaging in status/identity politics, not science. Additionally, you are in an argument where the evidence is vague enough that even experts in the field are disagreeing on which interpretation is more likely to be true; you are effectively telling at least one expert in the field "Despite being a lay member of the audience, I can tell that you are wrong." – medivh Jul 23 '13 at 13:33
12

In examining any claim, you always have at least two options: checking validity and checking the truth. In layman's terms, this means you can:

  • Examine the logical structure of the argument. Does the conclusion follow from the premises?
  • Examine the content of the argument. Are the premises in the argument true?

In your case, it seems you lack knowledge of the content, but anyone who understands logic should be able to—even with no understanding of the topic being discussed—critique the logical structure of an argument and check whether it is valid. Your examination would be entirely free from the influence of confirmation bias because logical coherence has nothing to do with content but of form. On the other hand, if you were to question the truth of the premises in the claim, you would be susceptible to (but not necessarily fall prey to) the confirmation bias, because the confirmation bias deals with the truth/falseness of information.

  • 1
    While this is a valid and true answer, could you be more specific in how this applies to questions of science, as opposed to claims in general? – Dave Liepmann Aug 25 '11 at 16:37
  • 1
    @Dave: Are you suggesting that scientific claims need a different set of criterion than "general claims" in order to be evaluated? – stoicfury Aug 25 '11 at 18:01
  • 1
    It seems to me that scientific claims are a subset of all claims, and that scientific claims require additional evaluation criteria than, say, logical arguments (which I think your answer addresses very accurately) or experiential/perceptual claims. Specifically I think that the contents of scientific arguments probably require additional consideration. – Dave Liepmann Aug 25 '11 at 18:12
  • 2
    That's an interesting notion. I personally wouldn't draw any distinctions in terms of additional consideration, even as a scientist myself. That is, strictly speaking I don't see any other grounds upon which to judge a claim, general or not. In practice, however, we can't always spend the time to rigorously go over each and every claim that appears before us, and thus we are left with something like what Michael describes in his answer, or (my personal favorite) W.K. Clifford's The Ethics of belief. – stoicfury Aug 25 '11 at 18:40
  • 1
    I am coming around to your point of view. Per this review of Mayo's work on the philosophy of science, I am coming to agree that "the distinction between what we call "science" and other sorts of reliable knowledge (or, if you like, other reliable practices of inquiry) does not reflect any deep methodological divide, but, say, is one of subject-matter, or even of the adventitious history of English usage". – Dave Liepmann Apr 20 '13 at 0:01
8

Even a seasoned research scientist will be qualified to directly evaluate experimental data only in a small subset of all areas. Fortunately, the peer review system allows scientists to make informed estimations regarding the research that falls outside of their area of expertise.

The best case scenario in science is that the experimental data is so overwhelming in support of one theory, that even those previously opposed cannot now justify opposition; the proper conclusion is totally obvious and consensus in the scientific community is ~100%. That doesn't mean that the theory is correct, but it does mean that anyone other than an expert possessing new experimental data or rare insight would probably be irrational or unable to doubt the theory supported.

Typically though, uncertainty centers around the questions that have not been firmly settled. In such a case, it is generally not appropriate even for an expert in the area to draw firm conclusions. The simplest heuristic in this case, is to assign a probability to each theory, where each probability is biased by the level of consensus in the scientific community, and keeping in mind that possibly none of the theories are correct. The more expertise you have in the area, the more you can use your own judgment to modulate those probabilities.

Often though, an issue is too new or fringe to have attracted the critique of the scientific community, or that critique has not reached the public. In this case, there are a few questions that should be asked by non-experts (keeping in mind that this list is non-comprehensive).

How much research is there in favor of the claim? does that research stand in opposition to or in the face of a large body of other research? Does it seem likely that, if the claim were true, it would have gone unknown until now?

Who is promoting the claim, through which forum, and for what potential motives? (If they're not scientists, or it's not published in a scientific journal, be suspicious. Also be suspicious of non scientists or scientist non-experts referencing obscure research in which they were not involved.)

Does the claim offer an easy solution to any of life's problems, and is it monetized by the people who back the claim.

Typically, the best sign that a claim is legit, is when the experimental evidence in favor continues to mount, originating from multiple independent laboratories, along with the approval of the scientific community.

  • I would add that unless you have personal stake in the problem, it is entirely acceptable to say "I have insufficient available evidence to pick a side in this debate." It is only necessary to pick a side when your choice of side will have an influence on your behavior (Regardless of degree of knowledge). – medivh Jul 23 '13 at 13:46
8

This is actually a very interesting epistemological question.

In Nyāya (classical Indian) epistemology, there are recognized four types of epistemological warrants (Pramāṇa-s):

  1. Perception
  2. Inference
  3. Analogy
  4. Authority

Now, with this in mind, let us take a case where you have a number of scientific studies with conflicting results, and you wish to form an opinion as to which is most reliable. Which of the aforementioned warrants could be used to lend credence to one or another study?

Clearly "direct perception" is unlikely, and from a scientific perspective would be likely to be viewed as anecdotal unless the observation was controlled, etc.,-- in other words, unless you were functioning as a scientist, which you stipulated was not the case.

Inference and Analogy could be useful for following individual claims, but at the same time, inferences and analogies are going to necessarily be based upon a process of reasoning from things you already believe to be true, which puts you in danger (as you point out) of confirmation bias.

Thus, oddly enough, we're left with basing our judgments upon the authority of others, and going with the study that comes from the source we judge to be most trustworthy. Which leads, of course, to the question of how we evaluate the qualifications of the authorities themselves-- which leads us back to the same list of epistemological warrants.

To cut to the chase: if you follow a foundationalist epistemology, you have some kind of axioms or first principles that you need to build up from (but it is likely that doing so would require you to actually become a scientist in order to make a decision in this case); if, on the other hand, you follow a coherentist epistemology, you adjudicate each new claim against the things you already believe to be true, which puts you in danger of confirmation bias.

Rock, meet hard place.

2

We know from experience that some questions are simple, some complex. What is the capital of Nebraska? What is 2+2? But others are more problematic: What is the cause of stomach ulcers? How many people must there be to guarantee that at least two of them have the same number of hairs on their heads?

Sometimes checking with our own eyes, thinking on that logically, being the scientist ourselves (but still not infallible in both sensation/method and judgement/logic) can lead to an answer, but sometimes we just don't have the resources to check; at some point we have to rely on the experience and judgement of others. At that point your own less expert judgement will be not about the original data but about seeing and judging the second hand and condensed information. All you have there is the experience of the contents of the report (which is not the data but a distillation of it) and the experience of the report itself, the author (reliability of similar reports by that scientist), even the publication method (the respect of the journal).

You'll have doubts about everything at every step (whether you are the original scientist or the second or third hand reader. But your experience (at each level) can help on judging reliability.

Is global warming human caused? Does evolution happen...sorry, does natural selection cause evolution? Even, does the moons path in the sky act on principles the same as a baseball? I don't have direct experience with any of these (except the baseball), but I can work with what others have told me so that I can 'suitably evaluate' them.

As to modern scientific journalism (reporting of scientific results in the current media), I am much more skeptical though; there's lots of (reported) bias, corruption, sensationalizing, story making.

So executive summary: yes, you as a non-expert can evaluate expert claims. It depends a lot on context (how much you already know, how good the report is); it's not as good as being omniscient, but it's better than nothing.

2

Yes, it's possible for a layperson to evaluate scientific disputes. The process, according to anthropologist Jennifer Raff, requires reading the science on the topic:

What constitutes scientific authority? ... What constitutes enough proof? Obviously everyone has a different answer to that question. But to form a truly educated opinion on a scientific subject, you need to become familiar with current research in that field. And to do that, you have to read the “primary research literature” (often just called “the literature”). You might have tried to read scientific papers before and been frustrated by the dense, stilted writing and the unfamiliar jargon. I remember feeling this way! Reading and understanding research papers is a skill which every single doctor and scientist has had to learn during graduate school. You can learn it too, but like any skill it takes patience and practice.

Her article, How to read and understand a scientific paper: a guide for non-scientists, describes this process in detail.

0

As Joseph has kindly pointed out, it depends completely on your level of knowledge in biochemistry and how much research and digging you are willing to do.

-2

In general, anyone who has studied Logic debate should be able to form reasonably sound arguments, and suss out weak arguments. I consider myself a laymen in a lot of things, but I feel with a couple hours of research and my knowledge of Logical debate I can participate in any argument and form strong arguments. That is not to say that they are undefeatable, as I cannot have researched everything in the field, and there may be evidence that I have not considered, but the arguments on their own are reasonably sound.

  • 1
    many fields (of ones I am familiar with: physics, math, computer science) take about 4 years (or more; usually this is called university) to get a lay person to the point of where they can make reasonably sound arguments and suss out weak arguments. Further, the question does not ask for these strong requirements of being able to actively participate in the scientific discussion, but just to evaluate a given claim for yourself. – Artem Kaznatcheev Sep 7 '11 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.