In recent times it has become more and more commonplace for journalists to present the results from studies which have not been published or peer-reviewed – without any reservations or warnings.

  • From a practical ethics standpoint is this something a journalist should do? What circumstances could justify it?
  • Is it even correct to describe such studies with phrases like “Researches at university X found that, ...”? Without peer-review there has not even been a basic sanity-check. In the case of an unpublished study, criticism is extremely difficult.
  • Should journalists avoid reporting about such studies alltogether? If not, how should they do it, considering their readers are mostly non-scientists? Probably some don't understand the significance of a study not being peer-reviewed or not even published.
  • I think the journalists should explicitly state the the study has not been published yet or is not peer reviewed. Though journalists have enough time presenting peer reviewed papers accurately anyways. – CognisMantis Mar 4 '17 at 1:05

We shouldn't put peer reviewed studies on a pedestal. The replication crisis unfolding in biomedical research, neuroscience, psychology, and other fields is largely about researchers trying to game the peer review system. At the same time, researchers in physics, biology, and social science frequently post working drafts on preprint sites, such as http://arXiv.org, http://biorXiv.org, and http://SSRN.com. Some of these sites are event experimenting with postpublication review, where reviewers post comments and recommend changes after the paper has gone public. (More here: http://www.the-scientist.com/?articles.view/articleNo/37969/title/Post-Publication-Peer-Review-Mainstreamed/)

In general, "researchers at university X found that ..." is going to overstate things, unless the "..." is about the particular observations that were made: "researchers at university X found that 7 of 13 participants preferred chocolate with peanut butter instead of jelly." Generalizations should be supported by several studies, preferably done by different groups of researchers over a period of time.

I would recommend that journalists emphasize methods and context rather than findings. In other words, first, science journalists should help readers understand how a study was done. Was this an experiment, a field study, or based on historical data? What techniques were used to produce the data, and how were the data analyzed? What assumptions did the study rely on? What limitations did it have?

Second, science journalists should help readers understand the scientific and social significance of the study. What other research has been done in the area? Does this study conflict or agree with that research? What disagreements do researchers in the field have? What arguments do they give for their views? Does the study have immediate commercial, policy, clinical, or social implications, or are the implications of the research more vague or long-term? (Very little "cancer research" is designed to directly produce new treatments.) If the research is socially controversial, what other factors ("besides the science") are contributing to the controversy?

  • As you say, "we shouldn't put peer reviewed studies on a pedestal." Check out ucpress.edu/book.php?isbn=9780520205130 The OP's real question is (or should be), how "to present the results from studies", whether peer-reviewed or not, with appropriate and easily-understandable caveats. – John Forkosh Mar 4 '17 at 5:05
  • but "unpublished" is even more serious. Isn't transparency part of science? How can an unpublished study even be distinguished from a private opinion of a few scientists? – wolf-revo-cats Mar 4 '17 at 7:55
  • I agree that unpublished research is a problem — there's very good evidence that "publication bias" or "the file-drawer effect" contributes to the replication crisis in psychology and problems with biomedical research. But publication on a "preprint" service, such as arXiv, is still publication. A journalist reporting on a preprint study should talk to other researchers in that field about the study — but they should be doing that anyways, even with peer-reviewed journal articles, as part of reporting the context of the study. – Dan Hicks Mar 4 '17 at 13:41
  • Maybe you're thinking about studies that have only been "published" as university press releases or conference posters. (I was focused more on the peer review part of your question.) I would agree that those should be treated skeptically. Without details of the methods used, it's pretty much impossible for anyone to assess the quality of the study. A journalist reporting on that kind of study can still report on context — what other work has been done, how the claimed findings might challenge the field, that kind of thing. – Dan Hicks Mar 4 '17 at 13:46
  • "Maybe you're thinking about studies that have only been "published" as university press releases or conference posters. (I was focused more on the peer review part of your question.)" – yes exactly. Especially those presented at conferences. The only way you get access is that the author mercifully sends you a copy. And it has become so common to report about the results of these studies, sadly not so much when severe flaws are found later and they don't get published. – wolf-revo-cats Mar 6 '17 at 0:30

Maybe it depends on whether the unpublished study is the kind of study that tends not to be published for bad reasons (e.g., it is a replication or failure to replicate rather than a novel finding).

If it's unpublished and it's the kind of study that tends not to be published for bad reasons, then journalists should talk about it. (But maybe the journalist should include something from the researchers whose finding(s) were supposed to replicate (so they can give their story about why their finding didn't replicate).)

If it's a study that tends not to be published for good reasons — e.g., substandard methods, it's a mess, etc. — then I'm not sure journalists should talk about it at all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.