Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020

Xavier Bouthillier, Gaël Varoquaux
Technical Report hal-02447823, 2020

PDF Code

How do machine-learning researchers run their empirical validation? In the context of a push for improved reproducibility and benchmarking, this question is important to develop new tools for model comparison. This document summarizes a simple survey about experimental procedures, sent to authors of published papers at two leading conferences, NeurIPS 2019 and ICLR 2020. It gives a simple picture of how hyper-parameters are set, how many baselines and datasets are included, or how seeds are used.


  title={Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020},
  author={Bouthillier, Xavier and Varoquaux, Ga{\"e}l},
  type={Research Report hal-02447823},
  institution={Inria Saclay Ile de France},