Home » Uncategorized » CHI 2023: Quick Look at How to Interpret Your Reviews

CHI 2023: Quick Look at How to Interpret Your Reviews

Julie R Williamson

Julie R. Williamson (Papers Co-Chair 2022 and 2023)

The first round of the review process is complete, and authors will now know if they have been invited to revise and resubmit (at least one reviewer recommends Revise and Resubmit or Better). This short post will help you understand how to interpret your reviews and decide if you want to revise and resubmit for CHI 2023.

In 2023, we received 3182 submissions.

  • 48.9% were invited to revise and resubmit
  • 43.7% were rejected after review
  • 7.4% were not sent for external review (withdrawn, desk rejected, quick rejected).

A significant proportion of papers invited to revise and resubmit will not be accepted to CHI 2023. Estimating based on previous years, we expect about half of the invited papers to be eventually rejected during the PC meeting in January 2023.

Our analysis of 2022 review process data demonstrates that papers that do not have strongly supportive reviews do not have a good chance to be accepted. This post will give some updated numbers to help interpret your reviews and decide if you want to participate in revise and resubmit. Authors do not need to notify the programme committee about their decision to revise and resubmit.

Review Scales

Before we go into the reviews, it’s important to remember what scales have been used during the CHI 2023 review process. Reviewers and ACs provide a recommendation (recommendation category out of 5 choices) and can further contextualise their recommendation based on originality, significance, and rigour (each a 5 point ordinal scale).

Recommendation

Short Name On Review Form Threshold for Revise and Resubmit
A I recommend Accept with Minor Revisions Yes
ARR I can go with either Accept with Minor Revisions or Revise and Resubmit Yes
RR I recommend Revise and Resubmit Yes
RRX I can go with either Reject or Revise and Resubmit No
X I recommend Reject No

Ordinal Scales

Ordinal scales are used to better contextualise reviewer recommendations, and should be considered as secondary to the reviewer recommendation.

Order On Review Form
5 Very high
4 High
3 Medium
2 Low
1 Very low

Proportion of Supportive Reviews

The proportion of supportive reviews (recommendations of RR or better) was a good indicator of paper success in 2022. Below, we provide bar charts showing a few ways of counting “proportion of supportive reviews.” In all cases, supportive means the actual recommendation of the reviewer, not the text of the review.

Figure 1 shows the proportion of reviewers recommending RR or better. Papers that have only one or two supportive reviews are very unlikely to be accepted and authors should consider if they want to revise and resubmit.

Figure 1. Proportion of reviewers recommending RR or better. Authors with at least 75% of reviewers recommending RR or better make up the top 32% of submissions.

Another way to look at how supportive reviews are is to consider how many reviewers recommend ARR or better. Papers where no reviewers recommend ARR or better are unlikely to be accepted.

Figure 2. Proportion of reviewers recommending “ARR” or better. Papers where at least half of the reviewers recommend “ARR” or better represent the top 14% of submissions.

A very positive way to look at how supportive reviews are is to consider how many reviewers recommend A, which is the most positive recommendation possible. However, 86% papers have no reviewers recommending A. This is an unfortunate statistic for our community, as our review process might be fairly criticized for being overly negative. This also means a fair number of papers where no reviewer recommends A in the first round will be accepted after the PC meeting in January 2023, showing the positive impact a revision cycle can have. It’s worth reflecting how we can improve the quality of submissions and the tone of reviews moving forward.

Figure 3. Proportion of reviewers recommending A. Papers where all reviewers recommend A are rare, representing just .5% of submissions.

Subcommittees

Each year, there is some variation between subcommittees. We provide this data for transparency and reflection on how different subcommittees are running their review process this year.

Figure 4. Breakdown of QR, X, and RR for all subcommittees.

Conclusion

This short overview of the review data should give you some additional context when analysing your reviews and deciding if you want to revise and resubmit. These give some indications, but all decisions are reached after discussion at the PC meeting. There are no deterministic positive or negative outcomes, all decisions are human decisions by the programme committee.

Good luck with your paper revisions, or if you are not revising and resubmitting, good luck with your future plans for your work. We hope the review process has provided something helpful in improving your papers and working towards your next publications.

Data Tables

Note some numbers may not add up to official totals due to conflicts, late reviews, and other missing data at time of writing.

Figure 1

Figure Description: Proportion of reviewers recommending RR or better. Authors with at least 75% of reviewers recommending RR or better make up the top 32% of submissions. See data table:

Rejected Revise and Resubmit
0.0% 1624 0
25% 0 246
50% 0 298
75% 0 473
100% 0 541

Figure 2

Figure Description: Proportion of reviewers recommending “ARR” or better. Papers where at least half of the reviewers recommend “ARR” or better represent the top 14% of submissions. See data table:

Rejected Revise and Resubmit
0.0% 1624 585
25% 0 538
50% 0 211
75% 0 118
100% 0 106

Figure 3

Figure Description: Proportion of reviewers recommending A. Papers where all reviewers recommend A are rare, representing just .5% of submissions. See data table:

Rejected Revise and Resubmit
0.0% 1624 1181
25% 0 287
50% 0 55
75% 0 19
100% 0 16

Figure 4

Figure Description: Breakdown of QR, X, and RR for all subcommittees. See data table:

QR (%) X (%) RR (%)
Accessibility and Aging A, Accessibility joint 0.0 0.463918 0.536082
Accessibility and Aging B, Accessibility joint 0.020833 0.375000 0.593750
Blending Interaction: Engineering Interactive Systems & Tools 0.025253 0.469697 0.505051
Building Devices: Hardware, Materials, and Fabrication 0.010638 0.329787 0.659574
Computational Interaction 0.016854 0.443820 0.533708
Critical and Sustainable Computing 0.033898 0.338983 0.627119
Design A, Design joint 0.033058 0.545455 0.421488
Design B, Design joint 0.016949 0.432203 0.550847
Games and Play 0.053435 0.473282 0.473282
Health 0.116505 0.451456 0.432039
Interacting with Devices: Interaction Techniques & Modalities 0.018433 0.470046 0.511521
Interaction Beyond the Individual 0.059603 0.417219 0.523179
Learning, Education and Families A, Learning joint 0.063158 0.463158 0.463158
Learning, Education and Families B, Learning joint 0.032609 0.543478 0.423913
Privacy & Security 0.068182 0.409091 0.522727
Specific Application Areas 0.046358 0.496689 0.456954
Understanding People: Mixed and Alternative Methods 0.113924 0.481013 0.405063
Understanding People: Qualitative Methods 0.078947 0.381579 0.539474
Understanding People: Quantitative Methods 0.074324 0.459459 0.466216
User Experience and Usability A, User Experience and Usability joint 0.081967 0.557377 0.360656
User Experience and Usability B, User Experience and Usability joint 0.049180 0.508197 0.442623
Visualization 0.046358 0.377483 0.576159
Facebook
Twitter
LinkedIn
Instagram