Peer-Reviewed Literature

Research is a collective activity that builds on the work of others and forms a foundation for other researchers, both now and in the future.

When research is concluded, it needs to be shared with the broader research community. But is also needs to be validated by the research community to assure it is meaningful and accurate.

The dominant process for such validation is peer-review, which involves submitting research reports to fellow researchers (peers) in that field of study for their comments and approval. Once an article has been accepted by peers, it is then published in academic journals that disseminate that information to the broader research community.

The peer-review process has a sequence of steps:

  1. Research: Experiments, field work, theoretical, modeling, etc.
  2. Writing: The research is documented in an article
  3. Submission: The researcher finds a journal that covers topics in the appropriate area of research and submits the article to the journal's editor
  4. Review: If the journal's editor feels the article might be a good fit for the journal, the editor will find two or three researchers who do the type of research described in the report. The reviewers will respond with one of four evaluations:
    • Accept: The article is ready to be published in the journal
    • Minor revisions: The article needs some minor revisions to be publishable
    • Major revisions: The article may have some merit but needs significant work to be suitable for the journal
    • Reject: The article has significant flaws and/or is unsuitable for the journal
  5. Revision: The authors make the changes suggested by the reviewers and/or explain why those suggestions are inappropriate. The article is resubmitted to the original peers for a second (or third) evaluation. In cases where the authors view the suggested revisions as unacceptable, an author may unsubmit their article and submit it to a different journal.
  6. Publication: Once the article is approved by the peers and the journal editor, it is published. Publication traditionally has meant printing on paper journals, but increasingly publication means publication on the journal's website.

Finding Peer-Reviewed Literature: Google Scholar

When performing a literature review, one resource that can be very useful is Google Scholar, a variant of the Google search engine that returns search results from peer-reviewed journals and other academic sources.

Google Scholar

Problems With Peer Review

While the peer-review process has, to some extent, been validated by the advance of science and technology over the past two centuries, it is not without significant flaws.

Faulty Research Slipping Through the Process

Peer-reviewers are not paid for their work. These reviewers are typically college faculty who already have significant workloads with their own teaching and research. Reviewers often do not have the time or ability to validate every result.

One notable case where faulty research slipped through the process was an 1998 article published in the prestigious medical journal The Lancet that tied autism to the MMR vaccine. After significant outcry from a variety of circles, the journal took the unusual step of retracting the article.

Wakefield etl al 1998

Academic Incentives Promote Quantity Over Quality

Academics (college faculty) are promoted based on their publication record, which is often measured by the number of articles they have published in major academic journals (research productivity). This leads to an emphasis on quantity rather than quality.

In 2013, the British physicist Peter Higgs, who discovered the Higgs boson, remarked that he could probably not get a significant academic position today because he published so few papers during his career. Further, he felt this impacted the quality of published work because, "It's difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964."

Peter Higgs (The Guardian)

The Science Reproducability Crisis

A 2016 survey reported on in the journal Nature found that 70% of researchers had tried and failed to reproduce another scientists experiments. Although less than a third of those surveyed felt this indicated that the results were wrong, this reflects a broader concern that many of the findings in peer-reviewed literature are faulty and cannot be generalized outside of the specific conditions that occurred during the research.

The Reproducibility Crisis (Nature)

Breakdown of The Peer-Review Business Model

Although researchers and reviewers are generally not paid to write and review for academic journals, publishing is an essential activity for academics in order to be hired, promoted, and given tenure.

There are expenses involved in publishing articles, and a number of profitable businesses have been built on academic publishing.

In the era when journals had to be distributed on paper, the journal companies made their money by selling subscriptions to academic libraries, at substantial cost. However, with the ubiquity of the internet and tightening budgets in many educational institutions, these subscriptions have proved onerous for college libraries, leading many libraries to drop non-essential subscriptions even as the demand by authors for publishing venues has grown.

This has led to the growth of open-access journals that are funded by article processing charges (APCs) paid by the authors themselves. This, in turn, has led to the growth of publishing environment where editors and publishers have an incentive to pass articles through a weak peer-review process rather than be selective about what articles get published or rigorous in validating the claims of published articles. It has also led to the rise of a whole class of predatory journals that exist to harvest APCs and provide questionable lines for the CVs of academics.

Beall's List of Predatory Journals and Publishers