Home » Submission Guides » Papers Review Process

Papers Review Process

Process Overview

We have refined the Revise & Resubmit process, which involves collapsing the two acceptance streams, and returning to a single PC Meeting for decisions, which makes the process a lot more similar to our previous rebuttal process, except that authors above the minimum threshold (see below) can revise (and rebut) over 5 weeks, rather than only rebut in a single week. These changes are summarized in the diagram below.

Image shows flow chart representation of the updated Revise and Resubmit process for CHI2023. The process spreads horizontally across reviewers, papers chairs, subcommittee chairs, associate chairs, and reviewers, and time progresses on the vertical, or between boxes on the horizontal. It starts with authors submitting papers, which are then checked by all three groups of chairs for different early reject conditions. If passing, reviewers are invited to review before a discussion between ACs and reviewers. Once the AC has written a meta review, the papers chairs release initial reviews to the authors. Any papers above a basic threshold (as per previous rebuttal years) are invited to revise and resubmit their paper; papers below the threshold are initially rejected, and no papers are directly accepted. ACs and Reviewers then re-review and discuss the revised papers and the author's response document. Then a single virtual PC meeting then takes place, before the papers chairs release final decisions to authors.

Desk Reject

Upon initial submission, the Papers chairs (PCs) and Subcommittee chairs (SCs) will determine whether the submission is appropriate for CHI or should be desk rejected. Desk rejects are made to save our reviewers’ time.

Examples of desk rejects are:

  • Incomplete submissions.
  • The paper lacks anonymization.
    • Leaving the author names in the paper or having a description or an acknowledgements section that reveals authors or the institution (e.g. places where user studies were conducted and specific supporting grant information).
  • Failure of declaration of concurrent submissions that are closely related to the submission.
    • If you have such a submission, you must include an anonymized version of that submission as a supplementary material. The same rule applies if your submission is built directly on a project described in a paper that is unpublished or currently under review at other venues.
  • Failure of declaration and/or citation of authors’ prior publications that are closely related to the submission.
  • Use of wrong formats.
    • All submissions must use the template specified in the Call for Submissions page. Note that CHI has used the single-column format since 2021, and submissions with other templates including double-column and extended abstracts will be desk rejected.
  • Clearly out of scope for the conference (e.g. formal methods for interstellar microcontrollers).
  • Not written in English.
  • Obviously not a conference paper (patent disclosure, popular press article, a complete book, etc.).
  • Something is so broken in the paper that makes it impossible to review.

Desk rejected submissions will not be assigned to a reviewer, and their authors will receive a brief note about the rejection.

Subcommittee and AC Assignment

For the submissions that are not desk rejected, SCs will check whether each of them has been submitted to an appropriate subcommittee with sufficient expertise to review. In rare cases, SCs will transfer papers that were initially assigned to their subcommittee to the secondary subcommittee chosen by the authors for more appropriate reviews. In a very exceptional case where the two subcommittees chosen by authors do not have the right expertise to assess the submission, the PCs will move it to another subcommittee.

All SCs and ACs are instructed to declare their conflicts of interests based on institutions and previous collaborations during the paper bidding phase. ACs are also instructed to express their preferences (or bids) on which submissions they would like to review. In this bidding phase, the title and abstract are the central information resources for ACs to identify submissions they prefer to review. Therefore, it is very important for authors to write titles and abstracts that clearly convey the core contributions of their submissions.

SCs will then perform paper assignments to Associate Chairs (ACs). ACs will be asked to complete paper bidding before this assignment so that their preferences and expertise are considered. Each submission will have two ACs, one as the primary AC (1AC) and the other as the secondary (2AC). 1AC is aware of the authors of the papers in order to best determine whether external reviewers are conflicted. The external reviewers are never aware of the authorship of papers. The identity of the authors is revealed to the secondary AC during the discussion phase of the process.

Quick Reject

The 1AC and 2AC will collaboratively check if a submission is a candidate for quick reject. The quick reject process entails considerable work for both ACs and SCs. Each paper will have at least 4 distinct, non-conflicted people (ACs, SCs, and PCs) that will have looked at any paper that is a potential quick reject, and all must agree that the paper would not have a chance of acceptance into the conference. The 1AC must first confer with the 2AC; come to an agreement and convince both SCs that the paper should be quick rejected. The paper will not require 2 external reviewers and thus decrease the overall load on the community. At this point, the 1AC must write up a sufficient and constructive review detailing why this paper is being quick rejected which must be acknowledged by the 2AC. This review will be checked by SCs.

This quick reject process sounds very harsh because it only provides a small amount of feedback. We brought it in because it allows us to offer rapid feedback to authors so that they can resubmit to other venues and also enables our reviewers to focus on submissions that have a high chance of improving and impacting the community.

Note that quick reject is NOT as easily determined by a few straightforward criteria, nor determined by a single person. Care is taken for a full justification of why this paper has not been reviewed by external reviewers. The burden on the ACs is particularly strong since they need to enter a complete and compelling review and the paper is still evaluated by both ACs and SCs (conflicts permitting).

The criteria for quick reject are:

  • HCI contribution is much too small given the length of the submitted paper – a long paper making a very small contribution may be quick rejected, as paper length should be commensurate with size of contribution.
  • The paper is too long (or too short) for a conference submission. The length of typical submissions is expected to be approximately 7,000–8,000 words excluding references, figure/table captions, and appendices. Submissions above 12,000 words or below 4,000 words may be quick rejected.
  • Grossly insufficient detail to replicate the apparatus or the experiment.
  • Grossly insufficient data to validate the analysis to support the claim.
  • Grossly insufficient literature review to contextualize and/or evaluate the proposed novelty/contribution to HCI in particular.
  • Paper is very sloppy: lots of typos, missing references, formatting issues (including large white spaces).

Round 1 Review Process

The 1AC will manage the review process and recruit two external reviewers to assess each submission that undergoes full review. The 2AC will provide a full review like the external reviewers. External reviewers should have high expertise on the topic of the paper(s) they are asked to review. As part of the review process Papers Chairs and Subcommittee Chairs reserve the right to get additional input from other ACs for any paper so as to make the best decision for acceptance. External reviewers will be asked to provide a detailed review of the submission and will be asked to provide a first round recommendation. Based on recommendations, papers will then fall into two groups:

Revise and Resubmit: Any paper that receives at least review recommending RR or higher will be given the opportunity to make changes and submit a track-changed version along with an author response to the reviews. If your reviews are positive, this may not involve many changes, but more borderline reviews may involve more work. Authors that meet this minimum threshold (at least one RR recommendation or higher, where RR/X is lower than RR) will have 5 weeks to respond as they see fit. Note: at this stage, no papers are Accepted, and we expect as many as 50% of R&R papers will eventually not be accepted.

Reject: Papers where all recommendations are X (Reject) or RR/X (between Revise and Reject) will be considered below the threshold to revise and resubmit, and so will be rejected. Note that unlike typical journals, the time frame for the Revise and Resubmit process is limited, and will not allow authors to execute another extensive implementation and/or evaluation or perform complete rewrite. A submission will thus be rejected if the reviewers feel that the required revisions would not be achievable within this short time frame.

SCs will ensure that, for each paper, all ACs and external reviewers will confer asynchronously to determine the first round, summary judgement.

Resubmission and Round 2 Review Process

Authors of the papers with the Revise and Resubmit recommendations are invited to submit a revision of their submission, and a response, by the resubmission deadline (see Important Dates). Revisions should use colors to highlight the changes to the document, and also include a response letter on how the reviewer comments have been addressed. The authors may communicate anonymously through the submission system with the 1AC for clarifications. If the 1AC does not respond, the authors should contact the PCs and SCs. No resubmission will be allowed after the deadline.

All the resubmissions will be reviewed by the original set of ACs and external reviewers. The 1AC reserves the right to invite new reviewers if needed. The results of the current review(s) and online discussions will be shared with the new reviewers in such cases. The result of the second round review is either Accept with Minor Revisions or Reject. Authors do not have a chance for another round of major revisions.

Facebook
Twitter
LinkedIn
Instagram