10 June 2026 | Tübingen

Improving the efficiency of training deep learning models

Cyber Valley Research Fund project develops more resource-efficient and autonomous training algorithms for neural networks

Frank Schneider and his research group have successfully completed their project “Resource-efficient autonomous training algorithms for deep learning” at the University of Tübingen. The project developed more resource-efficient and autonomous training algorithms for neural networks that have the potential to reduce training costs and improve model performance. Their findings are publicly available to both academic and industry labs, offering the potential for this research to be transferred into real-world applications.

The project was funded by the Cyber Valley Research Fund and was carried out between 2022 and 2024. It aimed to design training methods that could automatically adapt its hyperparameters (such as the learning rate) during the training phase of deep neural networks, reducing the reliance on user-defined or well-tuned hyperparameters. This would bring about a decrease in computational demands and energy consumption, making deep learning more accessible to researchers outside of the few top academic or industry labs.

An important outcome of the project was the development of the “AlgoPerf: Training Algorithms” benchmark: the first community-standard specifically designed to reliably measure the speeding up of neural network training resulting from algorithmic improvements. AlgoPerf was developed as an open collaboration within the MLCOmmons non-profit consortium, consisting of over 20 researchers from academia and industry, and led by two co-chairs: George Dahl (Google DeepMind) and Frank Schneider (University of Tübingen).

The project’s main achievement was to advance the evaluation standard for training methods, particularly those that do not require user-defined hyperparameters. All findings from this research project are publicly available, so that its algorithms and benchmark findings can be adopted by industry, potentially reducing training costs and improving model performance.

This project produced the following peer-reviewed publications:

Tatzel, L., Hennig, P., & Schneider, F. (2022). Late-Phase Second-Order Training. NeurIPS 2022 Workshop: Has it Trained Yet?. [open access]

Eschenhagen, R., Immer, A., Turner, R., Schneider, F., & Hennig, P.

(2023). Kronecker-Factored Approximate Curvature for Modern Neural Network

Architectures. Advances in Neural Information Processing Systems 36 (NeurIPS). [open access]

Kasimbeg, P., Schneider, F., Eschenhagen, R., Bae, J. et al. (2025). Accelerating neural network training: An analysis of the AlgoPerf competition. International

Conference on Learning Representations (ICLR). [open access]

Fernandez, A., Schneider, F., Mahsereci, M., & Hennig, P. (2025) Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods. Transactions on Machine Learning Research (TMLR). [open access]

About the Cyber Valley Research Fund

The Cyber Valley Research Fund was established to support Cyber Valley research groups undertake basic research in the fields of artificial intelligence and robotics. The fund totaled five million euros, including contributions from six of Cyber Valley’s founding corporate partners: Amazon, BMW, Bosch, IAV, Mercedes-Benz, Porsche, and ZF. It supported 20 research projects, the first of which began in 2020, and the final of which will conclude in 2026.