OPTIMIZATION OF HYPERPARAMETERS OF MACHINE LEARNING MODELS BASED ON EVOLUTIONARY ALGORITHMS AND SURROGATE MODELING
Relevance. Building high-precision predictive models in modern intelligent systems requires automating the search for optimal hyperparameters. Traditional optimization methods demonstrate low efficiency in high-dimensional spaces and with significant computational costs for estimating the objective function, which necessitates the development of new approaches at the interface of heuristic search and statistical approximation.
Problem. The main difficulty lies in the need to find the global extremum of the «black box» function with a strict limitation of the computing budget. The high resource intensity of each access to the complete machine learning model requires minimizing the number of iterations without losing the accuracy and robustness of the final solution.
Methods. A hybrid EA-SM algorithm is proposed that integrates the mechanisms of evolutionary search and adaptive surrogate modeling based on Gaussian processes. The mathematical apparatus includes the use of a data collection function to balance space exploration and exploit the found minima, as well as Tikhonov regularization to ensure the computational stability of covariance matrices.
Results. Experimental verification on the tasks of stochastic object classification and time series forecasting (AutoForecast, Chronos) confirmed the superiority of the method. A reduction in the number of calls to the objective function by 30-70% has been found compared to the DIRECT and Optuna algorithms, while maintaining high approximation accuracy in the vicinity of extremes.
Conclusions. The developed approach provides asymptotic convergence to the global optimum and resistance to stochastic noise. The algorithm is suitable for configuring neural network architectures in high-dimensional environments, minimizing time and hardware costs in monitoring and anomaly detection systems.
Zaripov E.A., Lazarenko S.A. Optimization of Hyperparameters of Machine Learning Models Based on Evolutionary Algorithms and Surrogate Modeling // Research result. Information technologies. – T.11, №2, 2026. – P. 109-120. DOI: 10.18413/2518-1092-2026-11-2-0-9
While nobody left any comments to this publication.
You can be first.
1. Akopov A.S. Modeling and Optimization of Individual Decision-Making Strategies in Multi-Agent Socio-Economic Systems Using Machine Learning // Business Informatics. 2023. Vol. 17. No. 2. pp. 7–19.
2. Anafiyev A.S., Karyuk A.S. Review of Approaches to Solving the Problem of Hyperparameter Optimization for Machine Learning Algorithms // Tavricheskiy Vestnik Informatics and Mathematics. 2022. No. 2(55). pp. 30-37.
3. Gorbunov S.M., Stanovov V.V. Evolutionary Algorithm for Multicriteria Optimization with Surrogate Machine Learning Models // Bulletin of the Bauman Moscow State Technical University. Series «Instrument Engineering». 2025. No. 2 (151). pp. 48–62.
4. Kleiner S.G. Study of the accuracy of the solution of the hyperparameter optimization problem using a neural network // Vestnik nauki. 2025. Vol. 3. No. 6 (87). Pp. 1785–1791.
5. Matveev A.N. Tools for constructing machine learning models // E-Scio. 2023. No. 6 (81). Pp. 71–77.
6. Timofeev A.V. Method for selecting hyperparameters in machine learning problems for classifying stochastic objects // Scientific and Technical Bulletin of Information Technologies, Mechanics and Optics. 2020. Vol. 20. No. 5. Pp. 667–676.
7. Trunov E. E. Algorithm for detecting anomalous behavior of users of automated systems based on machine learning methods // Bulletin of Astrakhan State Technical University. Series: Control, Computer Engineering, and Informatics. 2025. No. 4. pp. 33–42.
8. Usova M. A., Lebedev I. G., Shtanyuk A. A., Barkalov K. A. A global optimization algorithm for tuning hyperparameters of machine learning methods // Problems of Informatics. 2025. No. 4 (69). pp. 52–72.
9. Khodorchenko M. A., Butakov N. A., Nasonov D. A., Firulik M. Yu. A software framework for optimizing hyperparameters of topic models with additive regularization // Scientific and Technical Bulletin of Information Technologies, Mechanics, and Optics. 2023. Vol. 23. No. 1. pp. 112–120.
10. Abdallah M. AutoForecast: Automatic Time-Series Forecasting Model Selection // Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM ’22). 2022. P. 5–14.
11. Akiba T., Sano S., Yanase T., Ohta T., Koyama M. Optuna: A Next-Generation Hyperparameter Optimization Framework // Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019. P. 2623–2631.
12. Ansari A.F. Chronos: Learning the Language of Time Series // arXiv preprint arXiv:2403.07815. 2024. / Christ M. Time Series Feature Extraction on basis of Scalable Hypothesis tests (tsfresh – A Python package) // Neurocomputing. 2018. Т. 307. P. 72–77.
13. Audet C., Batailly A., Kojtych S. Escaping unknown discontinuous regions in blackbox optimization // SIAM Journal on Optimization. 2022. Т. 32. № 3. P. 1843–1870.
14. Candelieri A. Sequential model based optimization of partially defined functions under unknown constraints // Journal of Global Optimization. 2019. Т. 73. № 2. P. 281–303.
15. Conrad F. AutoML Applied to Time Series Analysis Tasks in Production Engineering // Procedia Computer Science. 2024. Т. 232. P. 849–860.
16. Filippou K., Aifantis G., Papakostas G.A., Tsekouras G.E. Structure learning and hyperparameter optimization using an automated machine learning (AutoML) pipeline // Information. 2023. Т. 14. № 4. P. 232.
17. Paulavicius R., Sergeyev Y.D., Kvasov D.E., Zilinskas J. Globally-biased BIRECT algorithm with local accelerators for expensive global optimization // Expert Systems with Applications. 2020. Т. 144. P. 113052.
18. Sergeyev Y.D., Kvasov D.E., Mukhametzhanov M.S. On the efficiency of nature-inspired metaheuristics in expensive global optimization with limited budget // Scientific Reports. 2018. Т. 8. № 1. P. 453.
19. Stripinis L., Paulavicius R. A new DIRECT-GLh algorithm for global optimization with hidden constraints // Optimization Letters. 2021. Т. 15. № 6. P. 1865–1884.
20. Sun Y., Yen G., Yi Z. IGD indicator-based evolutionary algorithm for manyobjective optimization problems // IEEE Transactions on Evolutionary Computation. 2019. Т. 23. № 2. P. 173–187.
21. Truong A. Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools // 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). 2019.
P. 1471–1479.
22. Wang H., Jin Y., Doherty J. Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems // IEEE Transactions on Cybernetics. 2017. Т. 47. № 9. P. 2664–2677.
23. Waring J., Lindvall C., Umeton R. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare // Artificial Intelligence in Medicine. 2020. Т. 104. P. 101822.
24. Xu N. Time Series Analysis on Monthly Beer Production in Australia // Highlights in Science, Engineering and Technology. 2024. Т. 94. P. 392–401.