Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020

How do machine-learning researchers run their empirical validation? In the context of a push for improved reproducibility and benchmarking, this question is important to develop new tools for model comparison. This document summarizes a simple survey about experimental procedures, sent to authors of published papers at two leading conferences, NeurIPS 2019 and ICLR 2020. It gives a simple picture of how hyper-parameters are set, how many baselines and datasets are included, or how seeds are used.

BibTeX:

@techreport{bouthillier:hal-02447823,
  title={Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020},
  author={Bouthillier, Xavier and Varoquaux, Ga{\"e}l},
  type={Research Report hal-02447823},
  institution={Inria Saclay Ile de France},
  year={2020},
  month={jan},
  url={https://hal.archives-ouvertes.fr/hal-02447823/file/ml_methods_survey.pdf},
  hal_id={hal-02447823},
  hal_version={v1}
}