Navigating the Meta-Review: Approaches and Considerations for NeurIPS 2019 Area Chairs

Overall, being an area chair for NeurIPS 2019 has been an interesting experience so far. The evolution of the reviewing process and the increase in submissions have definitely added a new layer of complexity to the role. As an AC, there are different approaches to take when it comes to the meta-review process. One option is to make the meta-review similar to an associate editor’s decision informed by reviews, responses, and discussions. This gives the AC more flexibility but may also introduce biases.

On the other hand, a panel summary approach, similar to the one used in NSF reviews, could provide a more objective and consensus-driven evaluation of the paper. This model involves a discussion led by a primary discussant and a neutral scribe who drafts a summary of the discussion for consensus. While this model may lead to more neutral reviews, it could also be challenging to implement due to varying levels of reviewer engagement.

Regardless of the approach taken, running the meta-review past the reviewers for feedback and edits could be beneficial in refining the final review. Ultimately, the goal of the meta-review is to provide a comprehensive and fair evaluation of the paper, incorporating discussions, responses, and diverse opinions. By finding the right balance between flexibility and objectivity, the AC can ensure a productive and effective reviewing process for NeurIPS 2019.