Hugging Face researchers attempted to replicate results from 2,200 papers presented at ICML. The effort highlights significant challenges in reproducing state-of-the-art machine learning models and training runs. This large-scale study provides empirical data on the current state of reproducibility in top-tier AI research.
- Large-scale reproduction attempts reveal systemic issues in replicating ML results.
- Missing code, hyperparameters, and hardware details are major reproducibility blockers.
- Even top conference papers frequently fail to reproduce their claimed performance.
- Standardizing reporting requirements could help improve transparency in AI research.
