"Many matters in academia, such as who gets funding, what gets published and who gets employed are settled by a relatively small group of people." (Lucas: "Legitimacy of peer review" )

Often we hear "peer-reviewed" praised as a feature of high-quality papers in science or mathematics - in particular in modern mathematics where a comparatively small fraction of believers in transfinite set theory have conquered most positions as academic decision makers.

This position is criticized by others, for instance [Henry Flynt: "Is mathematics a scientific discipline?" (1996)]

No matter how much the content of mathematics exploits paradox, mathematicians express dedication to policing their doctrine against inconsistency. Mathematicians do not welcome those who attempt inconsistency proofs of favored theories. [...] I will propose that the main factor in the establishment of "truth" in mathematics is professional procedure and discipline. [...]

Truth is negotiated on the basis of manipulation of import by distorting interpretations. Interpretation takes the form of discarding traditional intentions concerning mathematical structure: the privileged position of Euclidian geometry; the invariance of dimension; the association between integer and magnitude; uniqueness of the natural number series; etc.

From time to time, results are discovered which patently embarrass the conventional wisdom, or controvert popular tenets. [The Gödel theorems.] Then follows a political manipulation, to distort the unwanted result by interpretation so that it is seen to "enhance" the popular tenet rather than to controvert it. [...]

Even if my sense of the situation is right, the appearance of such a professionally compelling proof would be more a matter of packaging and selling than anything else. [...] The biggest hurdle such an attempted proof faces is professional discipline. Whether inconsistency proofs are recognized to have occurred is subject to entirely "political" manipulation.

On the other hand there are also peer-reviewed journals in other fields where the unscientific character is even more suspicious.

http://www.astrology-and-science.com/d-rese1.htm Contains abstracts of 91 studies, most of them empirical, from four astrological research journals. There are 37 abstracts from Correlation: Journal of Research into Astrology 1981-2007 published by the British Astrological Association, 22 from the now defunct Astro-Psychological Problems 1982-1995 published by Francoise Gauquelin in France and (in 1989-1990) by the National Council for Geocosmic Research in the USA, 18 from Astrologie in Onderzoek [Astrology under Scrutiny] 1986-2003 published by Wout Heukelom in the Netherlands, including its precursors 1977-1985 published by NVWOA the Dutch Society for Scientific Research into Astrology, and 14 from Kosmos 1978-1994 published by ISAR, the USA-based International Society for Astrological Research. At the time the first three journals were the world's only peer-review astrological journals devoted to scientific research, whereas Kosmos was more an astrological journal than a scientific research journal, hence the fewer abstracts.

So my question: Is "peer-reviewed" a guarantee of scientific quality or even truth or not, and if not, what ethical standpoint should be assumed and expressed toward it?

  • 1
    Maybe it's the converse that's relevant here: non-peer reviewed work is by definition beneath the threshold implied by consensus w/in scientific-community
    – Joseph Weissman
    Aug 20 '17 at 23:33
  • Peer-review is a well-known problem since it can go wrong. It's an attempt to maintain standards but it cannot guarantee quality or truth. It also tends to encourage a closed-shop. But nothing's perfect. Better to have peer review than a free-for-all. I'm not sure what the ethical point would be.
    – user20253
    Aug 21 '17 at 10:21
  • @PeterJ: In my opinion the ethical standpoint should include our attempts to make the disadvantages of the system public. My question was induced by the proud sentence of a believer in the system who boasted about proper peer reviewed research.
    – Heinrich
    Aug 21 '17 at 11:37
  • 1
    @Uwe I took the statement at face value, as a general remark, and wasn't taking sides. To the question, I'd say that peer-review is obviously not a guarantee of quality or truth or even, in some fields, scholarship and honesty, and so our ethical stance could be to approve of it as being better than nothing but to disapprove of its unreliability and tendency to fall foul of group-think. The tool is fine but the workmen could do better.
    – user20253
    Aug 22 '17 at 16:41
  • 1
    @PeterJ I fully agree with you, but I had the impression that some background information might be necessary to understand "Heinrich"'s reaction.
    – Uwe
    Aug 22 '17 at 17:02

I'm not sure what angle you'd like this question answered from, but presuming that, being a site about the study of philosophy, you're asking about the epistemic value of peer review as part of the wider "scientific method", then I think pretty categorically, no, "peer review" is not a guarantee of scientific quality.

Many people outside of academia misunderstand what peer review actually is, conflating it generally with "what other scientists think of the work". Peer review, as far as journal publication is concerned, is a very specific process involving a very small number of "peers" (usually three, only two of whom are independent of the publishing house), all of whom will have been appointed/selected by the editor of the journal, who in turn will have been appointed by the publisher, of which there are basically five (yes five), in the whole world. Thus five CEO's effectively exert their influence on the whole of academic research. More shocking is that the entire system was basically set up by Robert Maxwell as a money making enterprise.

The failure of this system to identify flaws in papers has been well documented by Ioannidis, Sokal, Smith, Bohannon.

Even the editors themselves have at time admitted the failure of the system, as Richard Horton of the Lancet put it

"The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability—not the validity—of a new finding…We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."

Ex-editor of the NEJM Dr. Marcia Angell

"It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines."

There is evidence of bias in terms of gender, race and institutional prestige have been found (8 out of 9 paper that journals had already published from prestigious institutions were rejected when researcher re-typed them and submitted them again with different institution names.

As a side note (nothing to do with the epistemic value of peer-reviewed knowledge claims), the power of peer-review does not stop at just what papers are read, but in England, it actually effects research funding via the government's RAE assessments (I think there's a new system now, but I've not been involved for quite some time), by which peer reviews of department performance on some very spurious criteria (like "impact") actually determines research grants to those departments.

As Nikolaus Kriegeskorte said in his submission to the Parliamentary review of the peer review system

"A scientific publication system needs to provide two basic services: access and evaluation. Access means we can read anything, evaluation means we don't have to read everything. The traditional publication system restricts the access to papers by requiring payment, and it restricts the evaluation of papers by relying on just 2‐4 pre‐publication peer reviews and by keeping the reviews secret."

He goes on to describe how Open-Access publications and the advances of modern technology can facilitate a more open and robust system which could actually further knowledge rather than restricts it. This was in 2011, to my knowledge absolutely none of the recommendations from any of the witnesses in that review have been adopted to date.

  • 1
    Where are you getting the three from? In my experience, it's usually been two (and rarely one!).
    – virmaior
    Aug 21 '17 at 12:13
  • 1
    @virmaior I was just thinking of the big five where there'd be typically two external peers and one internal (in house), I've made that clearer.
    – Isaacson
    Aug 21 '17 at 12:47
  • 1
    I'll go one step further and suggest that the credibility of peer-reviewed documents has probably declined in recent years. The war of propaganda is so intense, even many prominent "scientists" are examples of "controlled opposition" these days. I'm not going to trust a "peer" unless I can see his or her resume - and it had better be a good one. Aug 21 '17 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.