A peer is one who is of the same standing as another. Thus peer review is review performed by individuals considered to be professional equals.
In the scientific or scholarly peer review process, scholars’ manuscripts and grant proposals are reviewed and assessed by colleagues in their field. Reviewers are typically expected to evaluate the work’s:
Quality
Significance
Alignment with the goals of the journal or funding source
Nature & Function of Peer Review (cont.)
Peer review is considered to be "the cornerstone of the NIH [National Institutes of Health] extramural program in that it is the principal mechanism by which we identify high quality research that is worthy of funding" (Harold Varmus). This statement reflects the longstanding view, first expressed by Vannevar Bush in 1945, that the evaluation of science should be performed by scientists, not government officials.
Before the mid-1600s, scientists communicated through correspondence, the exchange of findings, and private printing of results. In 1665, the first scientific journals appeared and henceforth journal publication became central to the scientific enterprise. To ensure accuracy, accountability, and acceptability, submitted manuscripts were subjected to a pre-publication assessment. This represents the origin of the formal peer review process.
Nature & Function of Peer Review (cont.)
Serving as a peer reviewer is an important element of scholars’ participation in, and support of, their professional communities. Invitations to review manuscripts and proposals reflect one’s professional standing and recognition of expertise in one’s field. Peer reviewing is considered as a component of “professional service” in promotion and tenure evaluations.
Provide comments to an editor to help determine whether to publish a manuscript
Provide comments to a funding agency to help determine whether to fund a proposal
Provide constructive feedback that may help the author(s) improve the work
Peer review also serves to:
Reduce the number of errors in published work
Establish a level of quality, both perceived and actual, in the published record
Provide a fair, merit-based way to allocate limited journal space and funding
Nature & Function of Peer Review (cont.)
Peer review is not normally an effective mechanism for detecting fraudulent scientific practices, because the reviewer typically sees only the completed written document, not original data.
Peer review, however, sometimes leads to the detection of plagiarism, as reviewers who are intimately familiar with previously published work in the field are likely to recognize duplication. This is an incidental, but not unheard-of, function of peer review.
Peer Review Process: Manuscripts
Although each publisher has its own process for peer review of manuscripts submitted for publication, the following describes the basic process.
The manuscript is submitted to the publisher, usually via the editor responsible for a particular journal or area. The editor invites several potential reviewers to evaluate the manuscript. Reviewers are chosen primarily for their scholarly expertise relevant to the topic. Editors may identify potential reviewers based on:
Their own knowledge of the field
Consultation with colleagues
The list of works cited in the manuscript
Suggestions by the authors
In cases where there may be a conflict of interest or significant potential bias, authors may request that certain individuals not be asked to review their work.
Peer Review Process: Manuscripts (cont.)
After invited reviewers have confirmed their ability and willingness to evaluate the manuscript, the editor sends them a copy of the manuscript, specific instructions, and review criteria. This process, from initial invitations through submission of the review and notification of authors, is increasingly carried out entirely online through secure websites that protect the confidentiality of the materials and the process.
Completed reviews are returned to the editor, who then decides whether to:
Accept the manuscript
Reject the manuscript
Return the manuscript to the author(s) for revision and potential re-review
Editors are strongly influenced by the recommendations of their reviewers. Editorial decisions, however, at variance with the recommendations of external reviewers do occur.
Peer Review Process: Manuscripts (cont.)
Copies of the individual reviews (without identification) are usually included in the editor’s communication to the author(s) indicating the disposition of the manuscript.
If the editor recommends revision, the editor will often identify which of the reviewers’ concerns should be addressed prior to resubmission. The revised manuscript may be reviewed by the same or different reviewers.
Reviewers may be provided with anonymized copies of all comments, as well as informed of the final disposition of the manuscript. This is both a courtesy on the part of editors and a way to increase the value of the process for reviewers who can learn from one another’s perspectives and comments.
Peer Review Process: Manuscripts (cont.)
Various models exist regarding the identification of reviewers and authors to each other. In the physical and life sciences, the identity of
the author(s) is usually known to the reviewers, but the reviewers’ identities are withheld from authors. This is known as a
“single-blind” model.
Some journals send manuscripts to reviewers without identifying the author(s) (“double-blind” model). Manuscripts, however,
may contain enough clues (e.g., "Earlier work from our laboratory...") for reviewers to guess the identity of the senior author,
particularly since reviewers are familiar with the work of other research groups and individuals in their field.
A growing number of journals now employ a variety of "open peer review"
systems, where different aspects of the peer review system - reviewers' identity, reviewers' reports,
participation in the peer review process - have been made publicly available.
Journals regularly publish comprehensive lists of their reviewers as a way of giving them credit without associating their names with
specific manuscripts. Some journals may allow reviewers to opt to reveal their own identity to authors at the end of the review process.
Peer Review Process: Grant Proposals
Although the underlying goals and principles are the same, the logistics of the peer review process for grant proposals differs somewhat from that for manuscripts. Each funder or granting agency has its own process for peer review of submitted proposals. The following describes the basic process (specific steps vary among agencies) for organizations such as the National Science Foundation (NSF), NIH, and the National Endowment for the Humanities (NEH). When preparing a proposal, researchers should familiarize themselves with the review process as well as the funding criteria for the organization to which they are applying.
A proposal is submitted to the appropriate agency where it is assigned to a specific study section or review panel that will evaluate all proposals submitted in a specific area. NIH study sections and NSF review panels comprise both full-time agency personnel and selected individuals invited by the agency to participate in a particular review cycle. One or two members of the panel may serve as the main reviewers for the proposal.
Peer Review Process: Grant Proposals (cont.)
Individual proposals may also be sent to additional external or ad hoc
Ad hoc peer reviews are peer reviews for technical and scientific quality conducted by one or more expert and independent scientists outside a regularly scheduled panel.
reviewers; their reviews are then shared with all members of the panel evaluating the proposal.
At the review panel meeting, each proposal is assessed, and in some cases a priority score is assigned that will be used to determine funding. Some agencies use a second review body to examine proposals selected for funding to ensure that they are consistent with the overall mission of that agency. Others use additional review steps, such as the NEH, where all recommended applications are reviewed by the National Council on the Humanities and then the NEH Chairperson decides which applications will be funded.
Peer Review Process: Grant Proposals (cont.)
In most cases, researchers receive a summary of the review panel’s evaluation, whether or not the proposal is funded. Researchers may also receive copies of individual reviews. These documents frequently include suggestions or comments that can be very valuable in revising a proposal for resubmission or for submission to a different panel or agency.
A researcher whose proposal is not funded may discuss their concerns with the appropriate program officer. This is especially critical if the researcher plans to resubmit a revised proposal to the same panel.
Peer Reviewers’ Responsibilities
Integrity is a commitment to intellectual honesty and personal responsibility for one’s actions; this includes fairness in professional activities such as peer review. Accordingly, individuals who agree to conduct peer reviews have responsibilities with respect to:
Upon receipt of an invitation to review a manuscript or proposal, reviewers should:
Determine whether they are qualified to review the material
Determine whether they can complete the review by the stated deadline
Ensure that they have the complete document and review criteria
Examine the applicable criteria for conflict of interest which may affect their decision to accept an invitation to review a particular manuscript or proposal
Reviewers may also want to know whether:
The review process is anonymous
They will receive copies of other reviewers’ comments
They will be informed of the final disposition of the manuscript or proposal
Peer Reviewers’ Responsibilities (cont.)
When declining an invitation to review, individuals should:
Indicate their reason for not reviewing the document, for example
Insufficient expertise
Inability to complete the review by the deadline
Conflict of interest
Suggest alternative reviewers, if possible and appropriate
Individuals unsure about their appropriateness as reviewers should contact the editor or grant program officer for clarification.
Peer Reviewers’ Responsibilities (cont.)
Before reviewing manuscripts or proposals, reviewers should make sure they clearly understand:
General criteria upon which to base their review
Any specific issues they are asked to address
When their review is due, and how to submit it
Peer Reviewers’ Responsibilities (cont.)
The peer review system is dependent upon reviewers who bring a high level of professional competence to the review process. Peer reviewers should:
Read the proposal or manuscript thoroughly. Reviewers are often asked to provide a summary that represents their understanding of the critical issues in the document.
Perform a critical analysis of the manuscript or proposal
Adhere to the review protocol required by the journal or funding agency. Information is usually readily available online, for example
Follow any specific instructions from the editor or program officer, who may provide guidance on the process as well as the assessment criteria
Peer Reviewers’ Responsibilities (cont.)
In general, reviewers should distinguish comments about the manuscript itself (typically shared with authors) from recommendations
regarding publication (typically addressed only to the editor). All communication about the manuscript or proposal should be directed
to the editor, not the author(s).
The integrity of the peer review process requires that there be no conflict of interest, real or perceived, among reviewers and authors. The responsibility for identifying such conflicts rests primarily with individuals who are considering whether to accept an invitation to review.
In addition to conflicts based on individual relations (e.g., professional or personal association between reviewer and author) and on finances (e.g., reviewer is receiving funding from the same industrial source or a competitor), potential reviewers need to determine whether they have an intellectual conflict of interest (i.e., potential effect of the reviewer’s decision on their own work).
Peer Reviewers’ Responsibilities (cont.)
Peer reviewers should:
Be accurate. Errors, even honest ones, may cause harm both in substance (ultimate disposition of manuscript or proposal)
and in spirit (psychological).
Acknowledge their own professional limitations.
If the material is significantly out of their area of expertise, they should consider returning the document without
review, with a comment to that effect.
If they are able to evaluate some, but not all, aspects of the work, they should limit their comments to the areas in
which they are qualified, and clearly indicate that they have done so.
Peer Reviewers’ Responsibilities (cont.)
Peer reviewers should use careful judgment. For example, they should:
Distinguish significant concerns from personal preferences
Avoid applying higher standards to the work they review than to their own efforts
Strive to offer a balanced critique of the work, including both positive and negative aspects
Regardless of their opinion of the work, peer reviewers should be professional and respectful. Reviewers should not take advantage of their
anonymity to offer irrelevant criticisms or personal attacks, or use dismissive, categorical, or excessively negative language.
Peer reviews should be written in a professional manner and communicate personal respect for the authors, even when the
review identifies important flaws or significant issues in the work itself. Finally, reviewers should be fair in their reviews and not exhibit bias by applying
different standards based on unrelated characteristics of authors such as sex or national origin.
Peer Reviewers’ Responsibilities (cont.)
Peer reviewers should be as constructive as possible and take advantage of the opportunity afforded by the review process to help authors improve their work. For example, reviewers may suggest additional ideas, references, or analyses that might strengthen the manuscript or proposal. (It is then the authors’ decision whether to implement reviewers’ suggestions.)
By providing constructive, thorough, and clearly communicated evaluations of the work they have been asked to assess, reviewers fulfill their professional obligations to their individual peers and to their field as a whole.
Peer Reviewers’ Responsibilities (cont.)
A manuscript or proposal sent to reviewers is considered privileged communication and
confidential. Its contents should not be
shared with others without permission from the editor. If the reviewer feels the process would benefit from someone else’s input,
the editor must be consulted, and permission received BEFORE the manuscript or proposal is shared.
Misappropriation of unpublished manuscripts or proposals received for peer review may be considered theft of intellectual property. Examples include, but are not limited to:
Borrowing or quoting text
Using or copying information, such as data, images, or techniques contained in the work
Sharing, copying, or otherwise reusing any part of the document, including for teaching purposes
Concerns about using such technologies, include, but are not limited to:
The accuracy, consistency, and bias of the information generated.
Appropriate acknowledgement of source material in the output.
The tools’ lack of ability to reason or to understand meaning when generating output.
Tools possibly not including recent relevant information in the information generated due to outdated training data.
These concerns and others relate to the use of generative AI writing tools in the peer review process. Currently,
many publishers and funding agencies are determining whether these tools may be used in the peer review process since it has come to light that individuals have been using them:
some, such as the National Institutes of Health, have banned them.
Peer Reviewers’ Responsibilities (cont.)
Where generative AI writing tools are not banned, reviewers should be mindful of several of the core values of research when using them:
Accountability: Researchers are accountable for the reliability and veracity of their work. As peer reviewers, this includes issues of bias, misinformation, or errors resulting
from the use of AI writing tools in conducting reviews.
Honesty: If researchers use AI writing tools in their reviews, they must be honest about such use and disclose it so that any effects (negative or positive) can be assessed by others.
Dishonesty includes acts of omission (not revealing certain information) as well as commission, such as misrepresentation.
Openness: Researchers need to disclose how AI writing tools were used in the review process (e.g., name of the tool, the prompt, the date used, output).
Hosseini and Horbach (2023) recommend that if reviewers use generative AI writing tools to write scholarly reviews, they should “…disclose their use and accept full
responsibility for data security and confidentiality, and their reports’ accuracy, tone, reasoning and originality.”
Issues in Peer Review
The peer review process entails several types of challenges: procedural, conceptual, and ethical. Addressing these challenges requires a combination of practical/logistical and personal/ethical approaches.
Procedural challenges include ensuring that:
Manuscripts and proposals go to the appropriate individuals or review groups
Manuscripts and proposals from new professionals receive equitable treatment (avoiding favoring established researchers or an "old boy network”)
All participants recognize that reviewers’ errors, including those arising from personal biases, are harder to recognize
than errors made by authors or grant applicants
Issues in Peer Review (cont.)
In the past decade with the rise of electronic communication and computerized peer review systems, “fake” peer review or peer review
rigging has sprung up. In these scenarios, researchers set up email accounts for themselves and other colleagues with aliases and
recommend to journals these aliases as reviewers for their papers. Naïve editors then send manuscripts to them for review and researchers
review their own papers or those of a colleague. These reviews are usually quick and very positive to expedite the publication of a manuscript.
Peer review rings (or peer review and citation rings) are a similar mechanism using aliases and fake email addresses where a group of
researchers review and cite each other’s papers. Since the discovery of the loopholes in peer review software and systems,
many publishers put into place processes to address them and to prevent such exploitation in the future.
Issues in Peer Review (cont.)
Some ways to address procedural challenges are:
Including authors/proposal writers in the decision process for determining appropriate reviewers
Establishing clear guidelines for peer reviewers
Periodically assessing the reviewer pool and the configuration of review panels and how the selection of reviewers may influence review outcomes
Issues in Peer Review (cont.)
Conceptual challenges include:
Understanding the limits, as well as the strengths, of the peer review process
Recognizing that there is little information on how effectively the peer review process functions to select the best work for publication or funding, regardless of the criteria or procedures used
Some ways to address these challenges are:
Encouraging research on the peer review process itself from principles and practices to outcomes
Continuing to refine criteria for selecting and instructing reviewers in ways that will strengthen the integrity of the process
Issues in Peer Review (cont.)
Ethical challenges include:
Recognizing an inherent conflict in the peer review process: the individuals best qualified to serve as peer reviewers are also those most likely to benefit from privileged material in manuscripts and proposals.
Too-stringent application of guidelines for eliminating potential conflict of interest may weaken the peer review process, since the individuals who are best able to assess the work are also the most likely to be disqualified from doing so.
Some ways to address these challenges are:
Continuing to revise the guidelines for conflict of interest to achieve a workable balance between avoiding conflict and recruiting qualified reviewers
Recognizing the importance of trust among professionals and working to maintain that trust by teaching the rules and clarifying the ethical issues involved in professional practice for individuals entering the profession
Acknowledging that even though the system of peer review appears to work reasonably well, it is neither perfect, nor thoroughly assessed
Review Scenario 1
Scenario: Professor Jones is teaching a graduate seminar on grant writing and plans to cover the grant proposal review process, which is clearly an important topic for students to learn about. She recently participated in a National Institutes of Health (NIIH) review panel and still has copies of several of the proposals that were submitted, as well as comments, feedback, and ratings on those proposals. Reasoning that examining these proposals and the panel’s comments will offer invaluable “real-world” insights to students in the class, Professor Jones posts them on Canvas for students to read as a basis for class discussion. Which one of the following statements is correct?
Correct. This is correct. NIH policies on confidentiality in peer review are here
Incorrect. No, even though access will be restricted, it’s still a violation of NIH policies.
Incorrect. Even without authors’ names, it’s still a violation of NIH policies.
Incorrect. Any use of the proposals in this way is a violation of NIH policies.
Review Scenario 2 - Questions 1 & 2
Scenario: A graduate student meets with his advisor to discuss a new technique he may be able to apply to his research project. The advisor hands the student an unpublished manuscript that the advisor has just been asked to review as the manuscript includes detailed information about the technique that could be really helpful to the student. The advisor tells the student that while he’s free to read and use the information in the manuscript, he should not copy, share, or cite it, since it hasn’t yet been published. Which of the following statements are true?
1. Since the advisor told the student not to share the manuscript any further, there is no violation of confidentiality.
Incorrect.
Correct. The advisor showing the manuscript to the student is, itself, a violation of confidentiality, even if the student doesn’t share it with anyone else.
2. The advisor should not have shared the manuscript with the student.
Correct. The advisor showing the manuscript to the student is a violation of confidentiality.
Incorrect.
Review Scenario 2 - Questions 3 & 4
Scenario: A graduate student meets with his advisor to discuss a new technique he may be able to apply to his research project. The advisor hands the student an unpublished manuscript that the advisor has just been asked to review as the manuscript includes detailed information about the technique that could be really helpful to the student. The advisor tells the student that while he’s free to read and use the information in the manuscript, he should not copy, share, or cite it, since it hasn’t yet been published. Which of the following statements are true?
3. If the advisor feels it would be valuable for the student to read and perhaps help review the manuscript, the advisor can ask the author for permission to share it with the student.
Incorrect. Peer reviewers should communicate only with editors, not with authors.
Correct.
4. As long as the graduate student uses the manuscript only in this educational manner, it’s okay for the advisor to share it with him.
Incorrect. Any use of a manuscript in this way without permission from the editor is a breach of confidentiality.
Correct.
Review Scenario 2 - Question 5
Scenario: A graduate student meets with his advisor to discuss a new technique he may be able to apply to his research project. The advisor hands the student an unpublished manuscript that the advisor has just been asked to review as the manuscript includes detailed information about the technique that could be really helpful to the student. The advisor tells the student that while he’s free to read and use the information in the manuscript, he should not copy, share, or cite it, since it hasn’t yet been published. Which of the following statements are true?
5. If the advisor feels it would be valuable for the student to read and perhaps help review the manuscript, the advisor can ask the editor for permission to share it with the student.
Correct. Once such permission is formally granted, the student has the same rights and responsibilities as the advisor: that is, he may read and review the manuscript, though not share it or make use of the contents for his own purposes.
Incorrect.
Review Scenario 3
Scenario: Which of the following would be professionally appropriate comments to include in a peer review of a manuscript or a grant proposal (check all that apply)?
Correct. It states the reviewer’s opinion of strengths and weaknesses of the proposal. And if the funding program is looking for high-risk/high-reward projects proposed by young investigators, and cares more about investing in those individuals’ career development than in the specific research outcomes, this review could even support a decision to fund the proposal.
Incorrect. It sounds like this reviewer has a serious conflict of interest, and should not have accepted an invitation to review this paper. In any case, it’s unethical for a reviewer to seek personal or professional advantage by manipulating the process or attempting to hamper or slow down the work of competitors.
Correct. This is a reasonable and balanced way to communicate both a strength (interesting findings) and a weakness (poor writing) of the paper. It confirms that the work has value, but that it will require significant improvement to be publishable.
Correct. The reviewer clearly acknowledges and states the limits of the review, and evaluates only the portion of the work s/he is qualified to assess.
You have selected all of the correct statements.
Review Scenario 3 (cont.)
Scenario: Which of the following would be professionally appropriate comments to include in a peer review of a manuscript or a grant proposal (check all that apply)?
Incorrect. If you’re not an expert, you’re not qualified to review the paper, and merely opining that it “looks good” does not fulfill the criteria for peer review.
Correct. This is a clear, honest, and balanced response to a deeply flawed paper that nevertheless has some valuable content. Part of the reviewer’s job is to point out flaws; another part is to make constructive suggestions about what might be salvaged or reworked from a manuscript that should not be published in its current form.
Incorrect. The review should evaluate the work (in this case, the research design) – not the proposer’s need for funding.
Incorrect. This may be entirely true, but it’s an impolite and unhelpful way to communicate the reviewer’s opinion.
You have selected all of the correct statements.
Review Scenario 4
Scenario: Professor Brown is sent a manuscript to review. Unfortunately, it’s the middle of a busy field season and he is not going to be able to start on the review until very close to the due date. He’d like to review the paper though, and hopes he’ll be able to squeeze it in. He considers asking his postdoc to help since that way it’s more likely to get done on time. Which of the following statements are correct (check all that apply)?
Correct. Once such permission is formally granted, the postdoc has the same rights and responsibilities as the advisor: that is, he may read and review the manuscript, though not share it or make use of the contents for his own purposes.
Correct. If he doubts he can get it done on time, he should decline the invitation immediately so the editor can seek an alternative person to do the review. It would be helpful to the editor and to his own postdoc if the message declining the invitation also included a suggestion to invite the postdoc instead.
Correct. There’s no harm in asking. But then he’d better be absolutely sure to meet the extended deadline.
Correct. If he isn’t sure he can get it done on time, he should decline the invitation immediately so the editor can seek an alternative person to do the review. It would be helpful to the editor and to his own postdoc if the message declining the invitation also included a suggestion to invite the postdoc instead.
You have selected all of the correct statements.
Review Scenario 5
Scenario: A grant reviewer notices that some portions of the text in the project description are copied from earlier, published work in the field, but with no citations. The “borrowed” text provides an important part of the justification for the project which is, in fact, a very good one. But the reviewer is uncomfortable with the reuse of text in the proposal. Which of the following would be an appropriate response (check all that apply)?
Incorrect. This is a serious issue that compromises the work, and it is very important that it be brought to the program officer’s attention. The reviewer may be the only, or the first, person to notice the problem, so should definitely report it.
Incorrect. This is a serious issue that compromises the work, and it is very important that it be brought to the program officer’s attention. The reviewer may be the only, or the first, person to notice the problem, so should definitely report it.
Correct. This is a serious issue that compromises the work. It is very important that it be brought to the program officer’s attention
Correct. This is a serious issue that compromises the work, and it is very important that it be brought to the program officer’s attention
Incorrect. The reviewer should have no direct contact at all with the author, in the context of the peer review process.
You have selected all of the correct statements.
Case Study Review
Click on the image below to review the case study that was presented at the beginning of this module.
