Julie R. Williamson (Papers Co-Chair 2022 and 2023)
The first round of the review process is complete, and authors will now know if they have been invited to revise and resubmit (at least one reviewer recommends Revise and Resubmit or Better). This short post will help you understand how to interpret your reviews and decide if you want to revise and resubmit for CHI 2023.
In 2023, we received 3182 submissions.
- 48.9% were invited to revise and resubmit
- 43.7% were rejected after review
- 7.4% were not sent for external review (withdrawn, desk rejected, quick rejected).
A significant proportion of papers invited to revise and resubmit will not be accepted to CHI 2023. Estimating based on previous years, we expect about half of the invited papers to be eventually rejected during the PC meeting in January 2023.
Our analysis of 2022 review process data demonstrates that papers that do not have strongly supportive reviews do not have a good chance to be accepted. This post will give some updated numbers to help interpret your reviews and decide if you want to participate in revise and resubmit. Authors do not need to notify the programme committee about their decision to revise and resubmit.
Review Scales
Before we go into the reviews, it’s important to remember what scales have been used during the CHI 2023 review process. Reviewers and ACs provide a recommendation (recommendation category out of 5 choices) and can further contextualise their recommendation based on originality, significance, and rigour (each a 5 point ordinal scale).
Recommendation
| Short Name | On Review Form | Threshold for Revise and Resubmit |
|---|---|---|
| A | I recommend Accept with Minor Revisions | Yes |
| ARR | I can go with either Accept with Minor Revisions or Revise and Resubmit | Yes |
| RR | I recommend Revise and Resubmit | Yes |
| RRX | I can go with either Reject or Revise and Resubmit | No |
| X | I recommend Reject | No |
Ordinal Scales
Ordinal scales are used to better contextualise reviewer recommendations, and should be considered as secondary to the reviewer recommendation.
| Order | On Review Form |
|---|---|
| 5 | Very high |
| 4 | High |
| 3 | Medium |
| 2 | Low |
| 1 | Very low |
Proportion of Supportive Reviews
The proportion of supportive reviews (recommendations of RR or better) was a good indicator of paper success in 2022. Below, we provide bar charts showing a few ways of counting “proportion of supportive reviews.” In all cases, supportive means the actual recommendation of the reviewer, not the text of the review.
Figure 1 shows the proportion of reviewers recommending RR or better. Papers that have only one or two supportive reviews are very unlikely to be accepted and authors should consider if they want to revise and resubmit.
Another way to look at how supportive reviews are is to consider how many reviewers recommend ARR or better. Papers where no reviewers recommend ARR or better are unlikely to be accepted.
A very positive way to look at how supportive reviews are is to consider how many reviewers recommend A, which is the most positive recommendation possible. However, 86% papers have no reviewers recommending A. This is an unfortunate statistic for our community, as our review process might be fairly criticized for being overly negative. This also means a fair number of papers where no reviewer recommends A in the first round will be accepted after the PC meeting in January 2023, showing the positive impact a revision cycle can have. It’s worth reflecting how we can improve the quality of submissions and the tone of reviews moving forward.
Subcommittees
Each year, there is some variation between subcommittees. We provide this data for transparency and reflection on how different subcommittees are running their review process this year.
Conclusion
This short overview of the review data should give you some additional context when analysing your reviews and deciding if you want to revise and resubmit. These give some indications, but all decisions are reached after discussion at the PC meeting. There are no deterministic positive or negative outcomes, all decisions are human decisions by the programme committee.
Good luck with your paper revisions, or if you are not revising and resubmitting, good luck with your future plans for your work. We hope the review process has provided something helpful in improving your papers and working towards your next publications.
Data Tables
Note some numbers may not add up to official totals due to conflicts, late reviews, and other missing data at time of writing.
Figure 1
Figure Description: Proportion of reviewers recommending RR or better. Authors with at least 75% of reviewers recommending RR or better make up the top 32% of submissions. See data table:
| Rejected | Revise and Resubmit | |
|---|---|---|
| 0.0% | 1624 | 0 |
| 25% | 0 | 246 |
| 50% | 0 | 298 |
| 75% | 0 | 473 |
| 100% | 0 | 541 |
Figure 2
Figure Description: Proportion of reviewers recommending “ARR” or better. Papers where at least half of the reviewers recommend “ARR” or better represent the top 14% of submissions. See data table:
| Rejected | Revise and Resubmit | |
|---|---|---|
| 0.0% | 1624 | 585 |
| 25% | 0 | 538 |
| 50% | 0 | 211 |
| 75% | 0 | 118 |
| 100% | 0 | 106 |
Figure 3
Figure Description: Proportion of reviewers recommending A. Papers where all reviewers recommend A are rare, representing just .5% of submissions. See data table:
| Rejected | Revise and Resubmit | |
|---|---|---|
| 0.0% | 1624 | 1181 |
| 25% | 0 | 287 |
| 50% | 0 | 55 |
| 75% | 0 | 19 |
| 100% | 0 | 16 |
Figure 4
Figure Description: Breakdown of QR, X, and RR for all subcommittees. See data table:
| QR (%) | X (%) | RR (%) | |
|---|---|---|---|
| Accessibility and Aging A, Accessibility joint | 0.0 | 0.463918 | 0.536082 |
| Accessibility and Aging B, Accessibility joint | 0.020833 | 0.375000 | 0.593750 |
| Blending Interaction: Engineering Interactive Systems & Tools | 0.025253 | 0.469697 | 0.505051 |
| Building Devices: Hardware, Materials, and Fabrication | 0.010638 | 0.329787 | 0.659574 |
| Computational Interaction | 0.016854 | 0.443820 | 0.533708 |
| Critical and Sustainable Computing | 0.033898 | 0.338983 | 0.627119 |
| Design A, Design joint | 0.033058 | 0.545455 | 0.421488 |
| Design B, Design joint | 0.016949 | 0.432203 | 0.550847 |
| Games and Play | 0.053435 | 0.473282 | 0.473282 |
| Health | 0.116505 | 0.451456 | 0.432039 |
| Interacting with Devices: Interaction Techniques & Modalities | 0.018433 | 0.470046 | 0.511521 |
| Interaction Beyond the Individual | 0.059603 | 0.417219 | 0.523179 |
| Learning, Education and Families A, Learning joint | 0.063158 | 0.463158 | 0.463158 |
| Learning, Education and Families B, Learning joint | 0.032609 | 0.543478 | 0.423913 |
| Privacy & Security | 0.068182 | 0.409091 | 0.522727 |
| Specific Application Areas | 0.046358 | 0.496689 | 0.456954 |
| Understanding People: Mixed and Alternative Methods | 0.113924 | 0.481013 | 0.405063 |
| Understanding People: Qualitative Methods | 0.078947 | 0.381579 | 0.539474 |
| Understanding People: Quantitative Methods | 0.074324 | 0.459459 | 0.466216 |
| User Experience and Usability A, User Experience and Usability joint | 0.081967 | 0.557377 | 0.360656 |
| User Experience and Usability B, User Experience and Usability joint | 0.049180 | 0.508197 | 0.442623 |
| Visualization | 0.046358 | 0.377483 | 0.576159 |
