Importance of Code Submission in ICML 2019: A Look at Reproducibility and Transparency
ICML 2019 recently implemented an optional code submission policy for papers, sparking a conversation about the importance of reproducibility and transparency in machine learning research. As an area chair for the conference, I had the opportunity to review a mix of papers, some more theoretical and others more empirical, but almost all of them had some form of empirical validation. However, not all of these papers submitted their code along with their paper.
In the world of machine learning, where empirical validation plays a crucial role in making a case for a new method or algorithm, providing the code used in experiments is essential for ensuring reproducibility. Reviewers already approach papers with some level of skepticism, and without access to the code, it can be challenging to verify the results as described in the paper. By providing the implementation of the method, authors can strengthen their argument and make it easier for reviewers to trust the validity of their findings.
While there may be cases where providing code is not possible, such as in fields like electrical engineering where physical prototypes are used, in the world of machine learning, authors should strive to share their code whenever possible. Conferences like ICML are not as competitive outside of computer science, and the acceptance of a paper at a top conference can carry significant prestige. Authors benefit from the acceptance of their work by the research community, so it is only fair that they also share their implementations to contribute to the advancement of the field.
However, there may be cases where authors, particularly those from industry, may have proprietary code that they are not able to share. While this is understandable, it is important to consider the values of the research community and the benefits of open and free exchange of ideas. By encouraging, but not requiring, code submission, conferences like ICML are reinforcing the importance of sharing knowledge and techniques within the research community.
In conclusion, providing code along with research papers in machine learning is important for promoting transparency and reproducibility in the field. While there may be valid reasons for not sharing code, authors should strive to make their implementations available whenever possible to strengthen their argument and contribute to the advancement of the research community.