Posted on Leave a comment

Peer Review Primer-3: Outcomes

Why the sustained interest? Why do scientists consider peer review the bolster as well as the bane of their work? What does peer review research suggest would improve it? Ideally, at its core, the review of scientific research by knowledgeable peers provides scrutiny, musters arguments pro or con, and ultimately checks the integrity of the science as it passes through editors and publishing into the body of knowledge. At the same time and regardless of its form, the peer review process shapes the reputation and develops the character of the scientist as writer, as reviewer, and as editor. And, besides character, peer review is tied intimately to the ethics of scientific research and publication. For example, as recently as 2000, certain practices of authors paying to publish (except for extraordinary page charges) as well as suggesting peer reviewers was considered unethical. What else has changed to make formerly unethical conduct now acceptable or vise versa?  Ideally, then, authors, editors, and publishers would strive then to mentor and improve the practices that inhabit their respective communities. Why has peer review become so difficult?

Writing an editorial in the Journal of the American Medical Association last January (2018), Drummond Rennie and Annette Flannigan—current co-directors of the congresses—reviewed the history of the last thirty years of congresses and summarized the development of peer review research (see JAMA. 2018;319[4]:350-353. doi:10.1001/jama.2017.20606, https://jamanetwork.com/journals/jama/fullarticle/2670234 ).

Rennie and Flannigan acknowledge that a number of factors in the late 1980s period pointed to renewed interest and questioning of peer review. The rise of the Internet (starting in 1983) and the World Wide Web (launched by Tim Berners-Lee on August 1, 1991) became widely used for scholarly research, data exchange, and the print publishing process. Easy access and digital formats then combined with unscrupulous researchers and authors made it easier to do misconduct and harder to discover it through traditional formal peer review. Unregulated, the Internet provided both the ability for scientists to more widely share works-in-progress but also increased the opportunity for that work—from data, images, and articles—to be stolen, plagiarized, or falsified throughout the cycle of research, peer review and ever increasing illegitimate electronic publishing. This problem became so widespread that more than 4,000 “predatory journals”, questionable online-only publications solicited manuscripts, charged to publish, then uploaded papers without peer review (Jeffrey Beall’s list has been resurrected and is now maintained at https://beallslist.weebly.com/ ), At the same time, industry began to sponsor research with attendant, pro-business biases introduced—again, difficult for reviewers to detect.

Types of peer review changed in response to these “more” factors, but other tensions forced responses as well, often because the Internet provided less anonymity for authors and reviewers alike, less confidentially and privacy; forms of transparent peer review appeared. Publishers and authors sought efficiency and less time required before major discoveries or shifts in thinking became quickly disseminated; cascading peer review resulted. Open access both as a distribution concept and as a publishing model pushed print further into electronic forms; archives, especially university libraries, began publishing preprints, open reviews. Software was developed to address problems of access in these permanent digital archives (think DOI for publications and ORCID for people) and to help identify fraud. And the cycles rapidly recur, making it continually difficult and distressing for scientists and reviewers alike to keep pace. In academia, mentors and students also struggle with the ever-changing landscape of peer review in funding and in publishing. One enlightening way to take the pulse of peer review is to go back and sample abstracts, and now videos of presentations at the congresses—I often binge review an entire congress—especially if you cannot attend.

In her plenary session talk, “Peer Review: Are We Looking at the Right Things?” (https://www.youtube.com/watch?v=3KiJaIV3tg0) , on October 11, 2017 at the 8th International Congress, Lisa A. Bero defined quality in publication and discussed bias in the context of the various types of peer review emerging in today’s scientific publishing landscape. She also presented a neat summary of the problems and possible solutions in current peer review. One problem for which no solution was provided is the type of bias Bero calls “spin”; Bero has done extensive research on “spin” in scientific research reporting, (see http://sydney.edu.au/pharmacy/about/people/profiles/lisa.bero.php), especially in the area of drug trials. Bero is rightly concerned about the effect of selectively reporting research results to make them more acceptable to the author’s audiences, especially when that reporting favors drug company results and could possibly threaten patient welfare.

But “spin” often has been conflated with the more concise term “rhetoric of science”. “Bero defines spin in the above plenary talk as “biased presentation or interpretation to make results look more favorable” (min 8:36). Spin, therefore, is defined as conflicting with the core values of science. The core values of scientific research and publishing are objectivity, honesty, openness, fairness, accountability, and stewardship, according to the 2017 report of the National Academies of Science (see https://www.nap.edu/catalog/21896/fostering-integrity-in-research ). The rhetoric of science upholds these values when it is openly taught and understood.

My three-part blog comprising the primer on peer review also reflects those core values and often suggested ways to foster them through mentoring. When the Royal Society began publication in the seventeenth century, the core curriculum for learned men was grammar, logic, and rhetoric (the trivium) and arithmetic, geometry, music, and astronomy (the quadrivium). These subjects became known as the “liberal arts” education. The philosophical underpinning to the liberal arts education was that it created “good men living well”, in other words, the ideal was to educate people (some women were included) so that society could be rational and civil in its approaches to governing and public life. Consequently, the founders’ scientific education understood that communicating by public speaking and writing required an author to make choices, the best possible choices. to persuade the recipient of that message to receive then accept the communication. That is the essence of Aristotle’s definition of rhetoric: “Rhetoric then may be defined as the faculty of discovering the possible means of persuasion in reference to any subject whatever (Rhetoric, I. I. 14-11. 2,)*. In the very next sentence, Aristotle elaborates:

This is the function of no other of the arts, each of which is able to instruct and persuade in its own special subject; thus, medicine deals with health and sickness, geometry with the properties of magnitudes, arithmetic with number, and similarly with all the other arts and sciences. But Rhetoric, so to say, appears to be able to discover the means of persuasion in reference to any given subject.”

Our problem today with “spin” derives from, I believe, our global, marketing society as well as our naivety and ignorance about the role persuasion (i.e., rhetoric) has always had in science**. I have watched it grow into an ever-increasing problem for thirty years as fewer and fewer scientists received fewer liberal arts components in their education often which did not provide adequate instruction in the art of rhetoric. We are losing the ability to choose well when we write and speak or analyze other writing and speaking. We confuse honesty in reporting results with objectivity. We forget that science, the human quest for truth in the natural world is constrained by our ability to communicate what we discover. We forget that those who generate data know the most about it and deserve the first opportunity to interpret it well.  We abandon critical thinking when we think introductions and discussions—which authors use to help us see their inductive process and locate findings in the body of knowledge—are biased. We seek ways to make the science and its body of knowledge free from those human constraints without realizing that it is the good person, well educated, properly mentored, and with sterling character that makes good science from the bench to the journal article to the archive. We overlook and fail to reward education and mentoring in how to appropriately communicate science.

I have created the pyramids (see the Pyramids page of this website) to help scientists understand the rhetorical choices they must learn to master for their communications to be understood and for them to analyze the messages of their colleagues. (A video primer, downloadable model, and preprint on the method are also available on that page.) I teach using this method because it opens up the critical capabilities of scientists and instills awareness of what bias actually can be in their (and others’) communications. The model suggests that bias can be introduced by the scientists, the community supporting the science, the audiences to which it is directed, and the technology used to convey results. All these rhetorical elements that influence choices need to be studied. To understand the rhetoric used in science makes it possible to understand the power of truth and how easily it can be manipulated among those unaware.**

For example, I (and many others) have used one seminal paper of our time to teach, using a structured protocol, about rhetoric in scientific communication. James D. Watson and Francis Crick in “Molecular Structure of Nucleic Acids: A Structure for Deoxyribose Nucleic Acid.” (Nature 171, 4356 [25 April 1953]: 737-738. Available at https://profiles.nlm.nih.gov/ps/access/SCBBYW.pdf ) literally stole the show. Now archived with all the letters and papers related to that one-page paper in the Francis Crick Papers at the U.S. National Library of Medicine (https://profiles.nlm.nih.gov/ps/retrieve/Narrative/SC/p-nid/143/p-docs/true), the history and context of that article is usable. It is written in the first person! It selectively places the discussion among only six other papers as its authority! It plainly states that its methods will be published in another article! It plagiarized a stolen image of the helix and hints to its origins in the acknowledgments! It usurped first publication (by Oswald Avery, Colin MacLeod, and Maclyn McCarty on February 1, 1944 in the Journal of Experimental Medicine see https://www.sciencedirect.com/science/article/pii/S0960982213015157 )! It hints at how the DNA replicates but promises that disclosure in a subsequent article! What were the editors of Nature thinking? The article is clear, concise, and uses selected arguments throughout to persuade readers that this result of research was legitimate.

*Aristotle. 2000 (1926). The “Art” of Rhetoric. Trans. John Henry Freese. Cambridge, MA and London, England: Harvard University Press. p. 15. See that even Aristotle notes specific discourse to disciplines.

**Thomas S. Kuhn published twice on the topic of persuasion in scientific discourse. In 1970, The Structure of Scientific Revolutions (2nd ed. Chicago, IL: University of Chicago Press) and again in 1977 with The Essential Tension:  Selected Studies in Scientific Tradition and Change (Chicago, IL: University of Chicago Press.).

Leave a Reply

Your email address will not be published. Required fields are marked *