In response to the flood of very low-quality, often unreadable AI-generated papers, Computer Science venues have begun adopting stricter desk-rejection policies. For those unfamiliar, it used to be pretty hard to get a paper desk-rejected at most prestigious computer science venues… Everyone got their papers reviewed by three-ish people!
Clearly, this model isn’t scalable, as submissions have skyrocketed while the number of (reliable) reviewers has not. An apparently simple solution to this problem also does not work: having everyone be “forced” to review papers, regardless of their seniority status and previous experience with the venue, actually leads to appalling, lazy reviews. It can also cause a community to lose its identity way too quickly --- if a new group of academics starts to send papers to a given conference, they are just as important in shaping what appears in the conference as those who have built their careers on top of it. (This is a bit of a tangent, but a clear example is TheWebConference, or WWW, which was supposed to be a conference about the Web, but in practice is mostly a conference about ML methods loosely inspired by Web-related tasks…)
Enter the solution: conferences like CHI and USENIX or journals like TMLR have since adopted fairly strict, somewhat arbitrary desk rejection policies. The idea is that desk rejection should not only disqualify papers that are obviously doing something wrong (like those that are super long), but also that a more experienced researcher should take a cursory look at the paper and make an editorial judgment about whether this paper is worth people’s time---whether it has a meaningful chance assuming that reviewers will be serious about it. Even though conferences will probably avoid this term, they are proposing a “vibe check.”
This is a big shift from the typical role of an “area chair,” or whatever term is used in different computer science communities to denote something like the “handling editor.” Previously, this role has been exercised very gently --- this person who was supposed to do the meta-review and coordinate the discussion between the other reviewers was more of a mediator than a judge. They would question the reviewers and organize some discussion (or try to), but in a good number of cases, they acted as some sort of “review summarizer,” averaging out what the other reviewers said and creating a list of “asks” that ruled out the most outlandish requests. Now, these very people are asked to make a super consequential decision that will generally piss people off --- to desk-reject them!
People used to having their papers rejected through some sort of intricate process, where responsibility was shared among many parts (reviewer #2, the area chair, maybe some sort of committee meeting that is mentioned), are understandably upset that their papers are taken out of the game before they receive more attention. But this take underestimates that the attention pool we draw from is a relatively fixed, scarce resource. An area chair for CHI typically handles 14ish (!) submissions in a cycle! Meaning that a desk rejection policy that diminishes the incoming submissions by ~40% means that they will have something like 5 fewer papers a cycle to shepherd through full review!
But to judge whether this decision is good or bad, we must step back and think about what peer review is trying to accomplish. Confusingly, peer review has many goals! It should 1) be a mechanism for quality control, 2) a way to decide what deserves the community’s attention, and 3) an opportunity for authors to improve their work through expert feedback. My take is that increasing desk rejections sacrifices some of goal #3 to protect goals #1 and #2. Fewer authors will get detailed feedback, but reviewers have more time to carefully evaluate papers that make it through the cut. But in a second-order sense, desk rejections may also protect goal #3 --- as when everyone is overwhelmed with reviews, the expert feedback received by each individual author will inevitably suck.
And this does not mean that people won’t make mistakes in desk rejections --- I just had a paper desk rejected from CHI and, naturally, I think it was unfair! But after submitting to conferences where everyone is super burdened by reviews, I don’t think this other equilibrium really gives insightful feedback either! The hope is that with fewer papers to review, the ones that get through a senior person’s assessment will indeed receive the careful reading and thoughtful feedback that make peer review worth doing in the first place!
Think of a paper’s assessment as its underlying “merit” plus noise. In the no-desk-reject world, everyone is overloaded, and the noise impacts the whole review process. Desk rejection introduces another noisy decision easily on, but buys the papers that get through a more careful evaluation afterward. Whether this helps depends on the quality of the initial screening and how much attention it saves. My take is that senior researchers are actually reasonably good at knowing whether a paper is good, and that a ~40% decrease in reviewing will lead to way better reviews.
Some good papers will get an unfair desk rejection, and that sucks… But, in the absence of desk rejections, I believe that at least as many papers would get rejected without good feedback in a world of exhausted reviewers who are just writing lukewarm takes with ChatGPT. If we compare the desk-rejection process to the other review process we can realistically sustain, I think we probably will stick to the former --- and I say this even after being on the receiving end of a desk rejection I disagree with!