Posted on Leave a comment

Peer Review Primer-2: Evolution

From earliest examples of formal peer referees in the 1830s through to the 1970s, systems and processes that used peers to evaluate scientific papers reflected the composition of the societies and their disciplinary participants. Aileen Fyfe describes one such early set of members, those scientists of the Royal Society, as “all those involved in making editorial decisions in the 19th century were well-educated, affluent, white, British males, over the age of 40” (see https://blogs.royalsociety.org/publishing/peer-review-at-the-royal-society/  ). Fyfe acknowledges that the shared nature of the times, disciplines, and science probably fostered an “unconscious bias” that perpetuated over time to exclude women, people of color, anyone different from the process of both refereeing and publishing in science.  

Both informal and formal peer review processes require participants—authors, editors, and reviewers—to have characters with consistent honesty, charity, reliability, sincerity, legitimacy, and a working knowledge of disciplinary content, ethics, and that community’s communication practices.. For example, confidentiality, critiques, and suggestions traditionally were kept private among those who needed to know to protect authorship and copyright. Confidential letters between reviewers, editors, and scientists not only provided critique of the science, but inherently this writing presented documented arguments to publish. Study design and resultant data were maintained as a chain of evidence. Literature was cited as unique authority (published only once) to tie each study appropriately to the body of knowledge both in its introduction then through its conclusions. Publishers kept the archive accessible and protected. Authors, editors, and publishers had to balance pressures of time, cost, competition, reputation, against the innovation and contribution of the science. Part of retaining a positive reputation was making oneself aware of biases, disclosing them, understanding their effects, and taking care to mitigate those effects in the peer review. These same character traits apply to issues of authorship, data sharing, the ethical conduct of research, and publishing. Consequently, seeing the characters of the participants as important to the doing of science suggests that more care needs to be used in their mutual mentoring.

Moreover, when I teach graduate students scientific writing (journal article) and science writing (journalism) the process of practicing both informal peer review, and seeing their mentors do formal peer review, shapes such character traits in today’s scientific ecosystem. These character traits in turn help participants recognize and address ethical challenges, biases, and irregularities when those arise. In classical peer review, discovered irregularities remained in conversations between editors and reviewers then authors until resolutions were made or until institutions were required to be notified because participants assumed that honest mistakes could be made, discovered in peer or editorial review then rectified before publication. In informal peer review, participants learn to discuss differences, clarify writing and circumstances, and use an outlet’s protocol for peer review. Participants use what rhetorical analysis terms “conventions of the discipline” to become familiar with how their disciplinary communities communicate both formally and informally.

By the later 1970s, some combination of factors—personal biases, unscrupulous behaviors, unethical conduct, business pressures, funding processes, industry practices, institutional policies, science (and publishing) moving toward open access, and societal changes—pushed the processes of formal peer review out of the domain of science and forced it to straddle the domain of federal law and regulation. Science and its publishing became not just the activity of scientists but the target of institutions, businesses, and government, largely the funders of research. By the turn of the twentieth century, pressures of transparency and efficiency to change formal peer review yet again have brought it back around to open forms similar to informal peer review. The process and its evolution have been documented, but authors and editors also began to question whether peer review was living up to its purposes, why it was trusted yet despised, and if it was actually improving science. Peer review then became scrutinized by more detailed, designed study, that is, it became the focus of research.

Since 1989, eight international congresses have addressed research on scientific peer review and have been held, about every four years. The next one is scheduled to commence September 12-14, 2021, in Chicago, Illinois, USA. Presumably, the overarching theme of the Ninth International Congress on Peer Review and Scientific Publication is the same as the current tag line on the website, that is,  “Enhancing the Quality and Credibility of Science.”  Formerly, the biomedical sciences tended to dominate the congresses, but in 2018, the scope was broadened to include all sciences. One could spend months mining the content and reviewing videos or papers presented on peer review topics and research through these past congresses, available at https://peerreviewcongress.org/previous.html . I have summarized them briefly below.

The themes running through the congresses, and the papers presented relative to that theme, reveal both the ongoing positive results as well as problematic negative areas of research on peer review, Overall, the research focused on peer review participants, peer review process and external pressures on peer review. For example, processes of peer review in its many forms seems well documented; however, whether outcomes have improved science, peer review itself, or provided helpful innovations to better quality of review and of science in its process is less clear. Certain themes point to research on peer review participants—scientists and editors—like bias, spin, ethics, training, authorship and contributorship, persistent identifiers, incentives, and conflict of interest, Publishing and peer review research seems to be on its management: guidelines, types of review, products (preprints, article, and postprints), indexing, and archiving. External forces effecting peer review seem to be technology, costs, time, and forms of communication.

The responses to the research will be covered in the next blog.

Leave a Reply

Your email address will not be published. Required fields are marked *