5

Many matters in academia, such as who gets funding, what gets published and who gets employed are settled by a relatively small group of people. This applies quite broadly, from the arts to the sciences.

In the case of funding where public money is being spent on projects, a small group of people are making a decision on behalf of the population at large. The power to make these decisions is granted by the people in a very indirect way, elected representatives choose bodies, which then use a rather complex system of panels to make decisions. There are checks and balances of course.

But is this incredibly indirect representation of the population enough to make these panels legitimate? Are there other factors that legitimacy rests on?

Perhaps institutions responsible for making these decisions and the systems they employ provide a source of legitimacy. Perhaps its only the case when their procedures are out in the open.

Anyway, I'm struggling with this a bit, any suggestions would be welcome.

2

So there are problems with peer-review, but it's not quite as bad as you are making it out to be here.

Talking about the process of how a submission becomes published will help explain.

First, the author submits an article to a journal for consideration. At this stage, the article is prepared for blind review, so there is no identifying information in the manuscript.

Second, the editor of the journal will decide whether the submission is worth being refereed or not. If not, then the paper is "desk rejected". Many journals get several hundred submissions a year, and there simply aren't enough people to adequately referee them all. So the editor has to make some call about which papers are worth spending the valuable resources necessary to bring the article up to publication quality on.

If editor thinks the article maybe has a shot of getting published, he or she will send the article to referees for comment. Usually the paper goes to between one and three referees.

Different journals can be ranked in terms of who knows whom in this process.

A single blind journal will be one where the referees don't know the authors of the papers they are refereeing.

A double blind journal will be one where neither the referees nor the authors know each others identities.

A triple blind journal will be one in which neither the referees, nor the authors, nor even the editors know the identities of the authors and referees. (These kind of journals require editorial assistants who handle the emails and so forth to preserve the integrity of the process)

Now even triple blind journals are susceptible to failures of the review process. It can just so happen that you get asked to review a friend's paper, or a paper you happen to have heard at a conference. Ideally, a referee in such a position should recuse themselves, but my sense is that that doesn't happen often, just because it's often really difficult to find qualified referees who are willing to volunteer their time, so recusing oneself might be equivalent to getting the paper automatically rejected for lack of qualified referees.

There are problems with actual implementation to be sure, but overall I think the process at triple-blind reviewed journals is sound.

  • Hmm. "[I]n a survey of nearly 1,500 editors in chemistry, a plurality of respondents stated that double-blind was “pointless, because content and references give away identity”" (from the pdf in my comment to the question). Triple-blind isn't (even) mentioned, but that shouldn't make much of a difference. BTW: I don't know what 'a plurality' is supposed to mean here?? – user3164 May 9 '14 at 15:21
  • I guess they probably mean that the largest number, but a number less than 50% agreed with that statement. I'm a bit surprised by that finding, to be honest. – shane May 9 '14 at 15:25
  • You're right on plurality. I checked. Also: "This assumption has been tested in a number of empirical studies, which showed that reviewers can successfully identify authors 25%–40% of the time[.]" – user3164 May 9 '14 at 15:30
  • 1
    @shane I'm thinking of revising this question quite heavily, it will probably make this answer look a bit weird. I have a choice of whether to ask again, or edit. I thought I'd let you have a say in it. – Lucas May 9 '14 at 15:57
  • 1
    Just go on and revise it. If necessary we can revise my response, or hope somebody else comes up with a better one. – shane May 9 '14 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.