Surviving Peer Review

Photo by Clarisse Crosset on Unsplash

Handling tough feedback in the peer review process can be complex. In this article, I discuss two common challenging comments that are particularly relevant to machine learning research: implementing state-of-the-art methods and validating findings on larger datasets. Understanding why these requests arise and how you can navigate them will help you succeed as a researcher and computer scientist.

Why these comments are challenging

In computer science research, requests to acquire more data and compare a proposed algorithm against state-of-the-art (SOTA) models are among the most challenging reviewer comments, for several reasons:

1. Acquiring more data
   • collecting data can be time-consuming. In some cases (e.g., applied work in medical imaging or finance), ethical approvals or permissions can be required.
   • the additional data might introduce variability, making results less favorable or requiring further adjustments.
   • processing and analyzing new data adds a significant workload.
2. Comparing against SOTA models
   • some papers on SOTA models are not accompanied with code enabling full reproducibility, and some models require extensive hyperparameter tuning, making fair comparisons difficult.
   • there can be challenges due to computational costs, as deep learning models often require powerful GPUs and long training times.
   • lastly, the outcome can be negative: there is a risk that the SOTA method outperforms your approach, weakening your contribution.

Why these comments are formulated

Would a request to add more data or include a SOTA comparison suggest that your research is relatively weak on crucial aspects? Not necessarily:

• High standards for publication: many journals, especially top-tier ones, expect rigorous experimental validation, even for strong papers.
• Routine reviewer expectations: some reviewers always request more data or comparisons as a default without fully considering feasibility.
• A sign of interest, not rejection: if reviewers thought that your work was fundamentally flawed, they might have outright rejected it rather than suggested improvements. These comments suggest they see value but want stronger validation.
• Field-specific bias: in some fields (e.g., machine learning, medical imaging), it is standard to compare against deep learning, even if the study is not focused on it.

General tips

Here is some advice based on personal experience:

• Concerning open-source datasets, one source that I often use is Google Dataset Search.
• One may find common SOTA models and datasets in peer review articles. For instance, I have recently been especially interested in video forecasting models, and I found the following paper in TPAMI from Oprea et al. listing recent methods and data in that field. Being familiar with the academic literature in your field is key to understanding how your study compares and in which setting you have a greater chance to beat SOTA models.
• One can have a look at Papers with Code for SOTA models.
• Do not hesitate to use AI tools! ChatGPT, Perplexity, Claude... Although they are not perfect yet, they can help steer you in some positive direction.

Time management

The most difficult aspect of these comments is usually the time commitment that they require. It is easy to formulate criticism, but it can take weeks or months to address requests. Peer review in academia can be unpredictable: sometimes you may get relatively easy comments to answer, but sometimes, you will need to invest time more substantially.

• Make a timeline about how you wish to stay organized. If you do not have enough time, you can request a deadline extension.
• Ask yourself what you will sacrifice and reorganize your schedule (if needed) to deliver on your commitment to research and your career. A publication in a high-impact factor journal can open unexpected doors for you. Think about what motivates you in your research: what would be the benefits for the community if your work were to be published?
• Exploratory research before the initial submission may be the most enjoyable part of the process, handling reviewer comments, however, can feel like a race. At the resubmission stage, your day-to-day life may become more about (forced) execution than exploration. Consequently, you may need to place even greater emphasis on adopting the right habits (waking up early, doing exercise, eating well...) and execute on your plans.

One possible solution: partial revisions

If answering such requests feels impossible, one solution could be to request a partial revision. Unless the justification is strong, a partial revision is unlikely to be accepted in high-impact journals (typically with an impact factor greater than 5.0). Top-tier journals often enforce high experimental standards. However, the editor will make a decision at their own discretion. Some editors are flexible if you can convince them that the paper still represents a substantial contribution without the additional experiments. A partial revision could include the following:

1. Strengthening existing results – instead of adding new data, you could provide additional analysis on the current dataset (e.g., statistical analysis, visualization).
2. Theoretical or conceptual justifications – if adding SOTA comparisons is challenging, you could:
   • clearly state why your method differs from deep learning approaches (e.g., interpretability, lower computational cost, domain-specific advantages).
   • cite existing comparative studies that highlight deep learning's limitations in your context.
3. Acknowledging limitations transparently – journals appreciate honesty, so you can:
   • add a discussion about potential future work, stating that while deep learning comparisons would be ideal, they are beyond the scope due to computational or data constraints.
   • mention the results of some previous experiments where your method failed in particular settings.

Other challenging comments

Other reviewer concerns, such as insufficient novelty or unsupported conclusions, are also common but often require deeper methodological revision rather than tactical responses, and are therefore outside the scope of this article.

• michel.pohl@yahoo.com
• Oxford, United Kingdom