Peer-Reviewed Literature
Research is a collective activity that builds on the work of others and forms a foundation for other researchers, both now and in the future.
When research is concluded, it needs to be shared with the broader research community. But is also needs to be validated by the research community to assure it is meaningful and accurate.
The dominant process for such validation is peer-review, which involves submitting research reports to fellow researchers (peers) in that field of study for their comments and approval. Once an article has been accepted by peers, it is then published in academic journals that disseminate that information to the broader research community.
The Peer-Review Process
The peer-review process has a sequence of steps:
- Research: Experiments, field work, theoretical, modeling, etc.
- Writing: The research is documented in an article
- Submission: The researcher finds a journal that covers topics in the appropriate area of research and submits the article to the journal's editor
- Review: If the journal's editor feels the article might be a
good fit for the journal, the editor will find two or three researchers
who do the type of research described in the report. The reviewers will
respond with one of four evaluations:
- Accept: The article is ready to be published in the journal
- Minor revisions: The article needs some minor revisions to be publishable
- Major revisions: The article may have some merit but needs significant work to be suitable for the journal
- Reject: The article has significant flaws and/or is unsuitable for the journal
- Revision: The authors make the changes suggested by the reviewers and/or explain why those suggestions are inappropriate. The article is resubmitted to the original peers for a second (or third) evaluation. In cases where the authors view the suggested revisions as unacceptable, an author may unsubmit their article and submit it to a different journal.
- Publication: Once the article is approved by the peers and the journal editor, it is published. Publication traditionally has meant printing on paper journals, but increasingly publication means publication on the journal's website.
Finding Peer-Reviewed Literature: Google Scholar
When performing a literature review, one resource that can be very useful is Google Scholar, a variant of the Google search engine that returns search results from peer-reviewed journals and other academic sources.
Note that Google Scholar also includes dissertations, research reports, and working papers that are not peer-reviewed. While the information contained in a non-peer-reviewed document can still be useful and valid, it is not as trustworthy as information from an article that has gone through the rigorous vetting process required for publication in a peer-reviewed journal.
Therefore, you should be careful to identify whether an article found in Google Scholar is peer-reviewed before making it an integral part of your own research. Peer-reviewed literature is usually published on a web page that has something like Journal of... in the title.
Digital Object Identifiers
One challenge with peer-reviewed literature is that while scholarship is presumed to be a timeless contribution to the corpus of human knowledge, the internet where such literature is distributed is not. Beyond periodic changes in technology, all URLs on the internet are subject to link rot as links cease to work when publishers modify their business structures and / or reorganize their websites. This exacerbates an older problem of uniquely identifying articles among the chaos of similarly-named journals that are constantly coming into being or ceasing to exist.
One way to mitigate this identification challenge is use of the Digital Object Identifier (DOI) ® system. The DOI system was launched in 2000 to maintain a persistent registry of digital resources, including peer-reviewed articles. The DOI system is run by a foundation and operates under international standard ISO 26324.
A DOI consists of a numeric prefix identifying the registrar of an identifier (usually a publisher with peer-reviewed articles), followed by a suffix supplied by that registrant that uniquely identifies the object. For example, this is the DOI for Michael Minn's 2012 article, "The Political Economy of High Speed Rail in the United States" in the journal Mobilities:
10.1080/17450101.2012.655973
You can turn a DOI into a long-term URL by appending "https://doi.org/" to the start of the DOI. This link to the International DOI Foundation website will be updated to redirect to the publisher's current link for the article. You should generally try to use a DOI link rather than a publisher's link when writing a paper or report so that your links remain working over time.
https://doi.org/10.1080/17450101.2012.655973
You can usually find a DOI and/or a DOI link somewhere on the web page for a peer-reviewed article. If the article does not have a DOI, that is one indication that the article may not be peer-reviewed.
Problems With Peer Review
While the peer-review process has, to some extent, been validated by the advance of science and technology over the past two centuries, the peer-review process is not without significant flaws.
Faulty Research Slipping Through the Process
Peer-reviewers are not paid for their work. These reviewers are typically college faculty who already have significant workloads with their own teaching, research, and service. Reviewers often do not have the time or ability to validate every result.
One notable case where faulty research slipped through the process was an 1998 article published in the prestigious medical journal The Lancet that tied autism to the MMR vaccine. After significant outcry over both the questionable validity of the findings and the public health problems caused by the anti-vaccination movement, the journal took the unusual step of retracting the article.
Academic Incentives Promote Quantity Over Quality
Academics (college faculty) are promoted based on their publication record, which is often measured by the number of articles they have published in major academic journals (research productivity). This leads to an emphasis on quantity rather than quality.
In 2013, the British physicist Peter Higgs, who discovered the Higgs boson,
remarked that he could probably not get a significant academic position today
because he published so few papers during his career. Further, he felt this
impacted the quality of published work because,
"It's difficult to imagine how I would ever have enough peace
and quiet in the present sort of climate to do what I did in 1964."
The Science Reproducability Crisis
A 2016 survey reported on in the journal Nature found that 70% of researchers had tried and failed to reproduce another scientists experiments. Although less than a third of those surveyed felt this indicated that the results were wrong, this reflects a broader concern that many of the findings in peer-reviewed literature are faulty and cannot be generalized outside of the specific conditions that occurred during the research.
Breakdown of The Peer-Review Business Model
Although researchers and reviewers are generally not paid to write and review for academic journals, publishing is an essential activity for academics in order to be hired, promoted, and given tenure.
There are expenses involved in publishing articles, and a number of profitable businesses have been built on academic publishing.
In the era when journals had to be distributed on paper, the journal companies made their money by selling subscriptions to academic libraries, at substantial cost. However, with the ubiquity of the internet and tightening budgets in many educational institutions, these subscriptions have proved onerous for college libraries, leading many libraries to drop non-essential subscriptions even as the demand by authors for publishing venues has grown.
This has led to the growth of open-access journals that are funded by article processing charges (APCs) paid by the authors themselves. This, in turn, has led to the growth of publishing environment where editors and publishers have an incentive to pass articles through a weak peer-review process rather than be selective about what articles get published or rigorous in validating the claims of published articles. It has also led to the rise of a whole class of predatory journals that exist to harvest APCs and provide questionable lines for the CVs of academics.
