Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020

Xavier Bouthillier, Gaël Varoquaux
Technical Report hal-02447823, 2020

How do machine-learning researchers run their empirical validation? In the context of a push for improved reproducibility and benchmarking, this question is important to develop new tools for model comparison. This document summarizes a simple survey about experimental procedures, sent to authors of published papers at two leading conferences, NeurIPS 2019 and ICLR 2020. It gives a simple picture of how hyper-parameters are set, how many baselines and datasets are included, or how seeds are used.

BibTeX:

@techreport{bouthillier:hal-02447823,
title={Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020},
author={Bouthillier, Xavier and Varoquaux, Ga{\"e}l},
type={Research Report hal-02447823},
institution={Inria Saclay Ile de France},
year={2020},
month={jan},
url={https://hal.archives-ouvertes.fr/hal-02447823/file/ml_methods_survey.pdf},
hal_id={hal-02447823},
hal_version={v1}
}