Короткий адрес страницы: fornit.ru/71527 

Скопировано!

Относится к сборнику статей теори МВАП https://t.me/thinking_cycles

Peer Review of Scientific Works

There exist international guidelines and foundational works dedicated to the principles and methodology of peer review. 

• COPE (Committee on Publication Ethics) is the primary source on publication ethics. It includes:

  Code of Conduct and Best Practice Guidelines for Journal Editors
• Guidelines for Peer Reviewers

ICMJE (International Committee of Medical Journal Editors) includes sections specifically addressing the roles and responsibilities of reviewers.

Universally Accepted Principles of Scientific Peer Review Methodology:

1. Objectivity and Impartiality
   Reviewers must evaluate a manuscript solely on its scientific merit—not based on personal preferences, the author's reputation, institutional affiliation, or other irrelevant factors. Conflicts of interest must be disclosed, and reviewers should recuse themselves when necessary.

2. Scientific Rigor
   All critical remarks and recommendations must be supported by arguments: references to theories, empirical data, methodological standards, or logical inconsistencies in the text. Unsubstantiated subjective opinions are unacceptable.

3. Constructiveness
   The purpose of peer review is not to “reject” a manuscript but to help the author improve it. Even when serious flaws are identified, reviewers should suggest ways to address them or propose viable alternatives.

4. Adherence to Disciplinary Methodological Standards
   Reviewers assess whether the study complies with established standards in the field: appropriate sampling, methodological validity, adequacy of statistical analysis, and soundness of conclusions.

5. Respect for the Author
   Even when offering criticism, the tone must remain professional and respectful. Derogatory language, sarcasm, or condescension is unacceptable.

6. Confidentiality
   Reviewers must keep the manuscript content confidential and must not use the information for personal gain prior to publication.

7. Comprehensiveness of Analysis
   The review should cover all key aspects of the work:

   • Scientific novelty and significance
   • Logical structure and clarity of exposition
   • Methodological correctness and interpretation of results
   • Completeness and relevance of the literature review
   • Compliance with the journal’s formal requirements (where applicable)

8. Clarity of Expression
   Comments and suggestions must be formulated clearly, unambiguously, and specifically so the author can understand and act upon them.

    

General Principles of Peer Review Methodology

Above all, a scientific reviewer must possess a rigorous understanding of the principles of scientific methodology. Without this, they are not qualified to evaluate even a work within their own specialty. If a manuscript presents genuine novelty and the authors have invested significant effort in comparisons and generalizations, the reviewer typically lacks the time to independently construct an adequate mental model and compare it against their own expert prior assumptions. Even if the reviewer encounters something that contradicts their current understanding (which reflects the state of knowledge in the field), they usually lack the time to conduct proper verification.

1. Assess whether the methods and claims of the work comply with the universally accepted axioms of scientific methodology (not with postulated assumptions of the paper itself).
2. Assess whether any claims contradict the established axiomatic framework of the subject domain. 

• In the first case, if fundamental methodological principles are violated, the work may be deemed non-scientific. The reviewer should then return a comment requesting clarification—potentially allowing the authors to address the issue without invalidating their conclusions.

• In the second case, if the work challenges established domain axioms, the reviewer should request justification—specifically, experimental data or rigorous argumentation sufficient to warrant reconsideration of the prevailing paradigm.

   

In either scenario, the reviewer does not reject the manuscript outright but issues a “request for clarification” or “request for justification” to ensure conceptual precision. 

These two points constitute the most general principles of scientific peer review methodology. They subsume all other guidelines and capture everything essential to the review process. This approach provides a highly precise, two-step methodology for peer review.  

Principle 1. Assessment of Scientific Validity

• Essence: Evaluate whether the work adheres to the undisputed axioms of scientific methodology—logical consistency, verifiability/falsifiability, reproducibility, and causal justification.
• Reviewer Action: If a potential violation is identified, the reviewer does not reject the manuscript but issues a “request for clarity”, asking the authors to explain how their approach aligns with the violated fundamental principle.
• Outcome:
  • If the authors provide a satisfactory methodological justification, the manuscript proceeds to the next evaluation stage.
  • If not, the work is deemed methodologically flawed at a foundational level (i.e., non-scientific), and revision is only possible by resolving this core contradiction. 

Principle 2. Assessment of Alignment with the Knowledge Context

• Essence: Evaluate how convincingly the work justifies claims that contradict the expert consensus model (i.e., the established paradigm) of the field.
• Reviewer Action: Upon identifying such a contradiction, the reviewer does not dismiss it based on personal authority or conservative bias but issues a “request for justification”, requiring the authors to supply experimental evidence and arguments robust enough to warrant revising established views.
• Outcome:
  • If the authors present exhaustive and methodologically impeccable evidence, their innovative result gains credibility.
  • If the evidence is insufficient, the conclusions are downgraded to the status of hypotheses requiring further validation, or the manuscript is returned for revision. 

This approach claims to serve as the core ethical and intellectual code for scientific reviewers—especially crucial in an era of growing interdisciplinary, bold, and technically complex research, where traditional expert boundaries often prove inadequate. It ensures that conclusions are consistent and verifiable by other experts. 

Leveraging AI (e.g., GPT) in Peer Review

This methodology enables delegation of much of the analytical work to GPT-class systems, which possess the most comprehensive knowledge base up to their training cutoff. Failing to utilize this resource significantly increases the risk of overlooking critical issues. 

Unlike human reviewers, AI models lack “career inertia” or emotional attachment to paradigms—they strictly follow instructions. 

A recommended contextual prompt that already yields excellent results: 

Perform verification and assessment of compliance with the principles of scientific methodology. If any inconsistency with the undisputed axioms of scientific methodology is found, provide a corresponding remark.
Begin the evaluation from the perspective of a leading physicist. However, as soon as the work postulates or derives something that contradicts established views, do the following:

1. Assume it might be correct, and evaluate the internal logical coherence of the argumentation—not from the standpoint of orthodox beliefs, but based on the logical structure itself.
2. If the logic is sound, provisionally accept the claim as a postulate for constructing a framework of new representations.
   The final evaluation should not depend on how much the new theory contradicts the views of a leading physicist, but on the internal integrity and correct interrelation of the principles within the new theory. 

Strict Prompt (Eliminating GPT Discretion):

You are an intellectual assistant for methodological validation of a scientific manuscript. Your task is to verify compliance with fundamental principles of scientific methodology. 

Strict Rules (Non-negotiable):

1. Check the work against the following undisputed axioms:
   • Logical Consistency: Are there direct logical contradictions in the claims?
   • Verifiability / Falsifiability: Can the claims, in principle, be tested empirically?
   • Conceptual Clarity: Are all key terms clearly defined? 
2. If violations are found at this level, formulate a clear remark specifying the exact nature and location of the violation. 

Flexible Methodology (For Evaluating Novel Theories):

1. If the work proposes a theory contradicting established views, do not reject it on that basis alone.
2. Follow this algorithm:
   • Step 1: Deconstruction – Identify the core postulates of the new theory. Clearly distinguish between assumptions and logical consequences.
   • Step 2: Internal Validation – Assess whether the theory is internally consistent. Do the conclusions logically follow from the postulates without contradiction? Are all implications accounted for?
   • Step 3: Robustness Check – Analyze whether the authors’ arguments and data (if any) specifically support their postulate-based system, rather than alternative explanations.
   • Step 4: Comparative Analysis – Only after Steps 1–3 may you cautiously compare the predictive power or heuristic value of the new theory with existing ones—but frame this as a discussion point, not a verdict. 

Final Evaluation Criterion:
The work is methodologically sound if it passes the Strict Rules check and demonstrates internal logical coherence in Steps 2 and 3 of the Flexible Methodology. 

Your role is not to judge whether the theory is “true,” but to assess whether it constitutes a methodologically valid scientific hypothesis. 

Alternative Prompt:

You are a top-tier peer reviewer operating under the following protocol: 

Stage 1: Compliance with Universal Scientific Methodology Principles
— Evaluate logical consistency, operational clarity, falsifiability, reproducibility, and validity of conclusions drawn from data.
— Any violation constitutes a critical flaw. Such work is deemed non-scientific, regardless of content. 

Stage 2: Novelty Assessment within the Subject Domain
— If the work contradicts established knowledge, do not reject it immediately.
— Switch to the mode: “Suppose this is true. What postulates and consequences would follow?”
— Then evaluate:
a) Internal coherence of the new theory,
b) Sufficiency of argumentation (logical or empirical),
c) Potential for integration or comparison with existing data.
— If the theory is methodologically sound, provisionally accept its postulates for the purpose of analysis and judge the work by its internal integrity—not by its deviation from orthodoxy. 

The final assessment depends not on how “familiar” the work appears, but on how scientifically rigorous its form is and how well-substantiated its content is. 

The human reviewer’s role then reduces to verifying the AI-generated output, offering expert judgment, and requesting clarifications from the authors if needed. 

This methodology also enables the inclusion of a supplementary statement in scientific manuscripts, alerting reviewers to these principles and facilitating their work. Example of such an insert: 

To counteract the Semmelweis effect (fornit.ru/68139) and confirmation bias (fornit.ru/68578), a two-point methodology exists for evaluating the scientific validity and soundness of a manuscript. This approach incorporates all universally accepted ethical and scientific peer review principles and minimizes review time—even when the reviewer is not a specialist in the specific subject area: fornit.ru/71526 . 

Additional discussion on scholarly opposition in this context: fornit.ru/71498 . 

Nick Fornit