PARAMETER TUNING IN BACKPROPAGATION NEURAL NETWORKS: IMPACT OF LEARNING RATE AND MOMENTUM ON PERFORMANCE
DOI:
https://doi.org/10.33480/jitk.v11i1.6484Keywords:
accuracy optimization , artificial neural networks, backpropagation , learning rate , momentumAbstract
Artificial Neural Network (ANN) play a pivotal role across diverse domains, including medicine, economics, and technology, due to their ability to model complex relationships and deliver high prediction accuracy. This study systematically examines how learning rate and momentum interact in backpropagation, moving beyond isolated analysis to enhance ANN performance. A qualitative research design employing a systematic literature review was utilized, with data sourced from reputable databases covering the past 11 years. Bibliometric tools such as VOSviewer and R-Studio were applied to identify trends and patterns in the literature. The findings reveal that both learning rate and momentum significantly impact convergence efficiency and model stability. Backpropagation remains fundamental for weight adjustment in minimizing prediction errors. ANN optimization demonstrates substantial practical benefits, including enhanced treatment outcome predictions in medicine, modeling nonlinear patterns in economics, and improved image classification accuracy. However, challenges such as the curse of dimensionality, overfitting, and dependence on large datasets persist. Strategies such as regularization, ensemble methods, and sensitivity analysis present viable solutions. This study underscores the critical need to advance ANN optimization techniques and highlights the potential of interdisciplinary collaboration in addressing existing limitations and broadening ANN applications
Downloads
References
Vidya Chandgude and Bharati Kawade, “Role of Artificial Intelligence and Machine Learning in Decision Making for Business Growth,” Int. J. Adv. Res. Sci. Commun. Technol., vol. 3, no. 1, pp. 54–58, 2023, doi: 10.48175/ijarsct-8556.
T. Ahmad, R. Madonski, D. Zhang, C. Huang, and A. Mujeeb, “Data-driven probabilistic machine learning in sustainable smart energy/smart energy systems: Key developments, challenges, and future research opportunities in the context of smart grid paradigm,” Renew. Sustain. Energy Rev., vol. 160, p. 112128, 2022.
M. G. M. Abdolrasol et al., “Artificial neural networks based optimization techniques: A review,” Electronics, vol. 10, no. 21, p. 2689, 2021.
R. Abdulkadirov, P. Lyakhov, and N. Nagornov, “Survey of optimization algorithms in modern neural networks,” Mathematics, vol. 11, no. 11, p. 2466, 2023.
N. F. Hidayanti, Syaharuddin, N. H. I. Ningsih, A. Hulaimi, Z. Ariani, and D. Iswanto, “Prediction of Macroeconomic Growth Using Backpropagation Algorithms: A Review,” Int. J. Sci. Res. Manag., vol. 13, no. 01, pp. 8255–8266, 2025, doi: 10.18535/ijsrm/v13i01.em07.
Syaharuddin, Fatmawati, H. Suprajitno, and Ibrahim, “Hybrid Algorithm of Backpropagation and Relevance Vector Machine with Radial Basis Function Kernel for Hydro-Climatological Data Prediction,” Math. Model. Eng. Probl., vol. 10, no. 5, pp. 1706–1716, 2023, doi: 10.18280/mmep.100521.
V. B. Parthasarathy, A. Zafar, A. Khan, and A. Shahid, “The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities,” arXiv Prepr. arXiv2408.13296, 2024.
C.-H. Chen, J.-P. Lai, Y.-M. Chang, C.-J. Lai, and P.-F. Pai, “A study of optimization in deep neural networks for regression,” Electronics, vol. 12, no. 14, p. 3071, 2023.
C. Song, “The performance analysis of Adam and SGD in image classification and generation tasks,” Appl. Comput. Eng., vol. 5, pp. 757–763, Jun. 2023, doi: 10.54254/2755-2721/5/20230697.
R. Ando, Y. Fukuhara, and Y. Takefuji, “Characterizing Adaptive Optimizer in CNN by Reverse Mode Differentiation from Full-Scratch,” Indian J. Artif. Intell. Neural Netw., vol. 3, no. 4, pp. 1–6, 2023.
E. Rybko, E. Voevodina, and A. Burykin, “POTENTIAL OF USING NEURAL NETWORKS IN THE FIELD OF MEDICINE,” SOFT Meas. Comput., vol. 11–2, pp. 39–45, Jan. 2022, doi: 10.36871/2618-9976.2022.11-2.004.
D. Xhako and N. Hyka, “Artificial neural networks application in medical images”.
D. Jin and J. Xu, “Artificial Neural Networks and Its Applications in Chemical Industry,” Asian J. Chem. Sci., pp. 31–39, Nov. 2022, doi: 10.9734/ajocs/2022/v12i3221.
R. Weiss, S. Karimijafarbigloo, D. Roggenbuck, and S. Rödiger, “Applications of neural networks in biomedical data analysis,” Biomedicines, vol. 10, no. 7, p. 1469, 2022.
A. Daydar, “Development of Effective Artificial Neural Network Model using Sequential Sensitivity Analysis and Randomized Training,” Int. J. Soft Comput. Eng., Jul. 2021, doi: 10.35940/ijsce.f3515.0710621.
E. P. Onakpojeruo and N. Sancar, “A Two-Stage Feature Selection Approach Based on Artificial Bee Colony and Adaptive LASSO in High-Dimensional Data,” AppliedMath, vol. 4, no. 4, pp. 1522–1538, 2024, doi: 10.3390/appliedmath4040081.
V. Mandailina, A. Nurhalimah, S. Mehmood, Syaharuddin, and Ibrahim, “Study of Climate Change in the Mandalika International Circuit Area Using Neural Network Backpropagation,” Rev. d’Intelligence Artif., vol. 36, no. 6, pp. 847–853, 2022, doi: 10.18280/ria.360604.
M. Z. Islam, M. M. Abdul Kader Jilani, and M. R. Karim, “Enhancing post-training evaluation of annual performance agreement training: A fusion of fsQCA and artificial neural network approach,” PLoS One, vol. 19, no. 6, p. e0305916, 2024.
I. Masic, “Scientometrics: The Imperative for Scientific Validity of the Scientific Publications Content,” Sci. Art Relig., vol. 1, no. 1, pp. 56–80, 2022, doi: 10.5005/jp-journals-11005-0017.
S. Srimamilla, “Image Classification Using Convolutional Neural Networks,” Int. J. Res. Appl. Sci. Eng. Technol., vol. 10, pp. 586–591, Dec. 2022, doi: 10.22214/ijraset.2022.47085.
A. P. W. Solikhun and P. Alkhairi, “Bone fracture classification using convolutional neural network architecture for high-accuracy image classification,” Int. J. Electr. Comput. Eng., vol. 14, no. 6, pp. 6466–6477, 2024.
M. Ashawa, N. Owoh, S. Hosseinzadeh, and J. Osamor, “Enhanced Image-Based Malware Classification Using Transformer-Based Convolutional Neural Networks (CNNs),” Electronics, vol. 13, no. 20, p. 4081, 2024.
J. Saxena and A. Nagraj, “An Optimized Technique for Image Classification Using Deep Learning,” Int. Res. J. Comput. Sci., vol. 10, pp. 97–103, Jun. 2023, doi: 10.26562/irjcs.2023.v1004.11.
X. Wang, M. Du, A. Zhang, F. Li, M. Yi, and F. Li, “Classification and Recognition of Doppler Ultrasound Images of Patients with Atrial Fibrillation under Machine Learning,” Sci. Program., vol. 2022, no. 1, p. 4154660, 2022.
T. E. Anju and S. Vimala, “Ensemble Residual Network with Iterative Randomized Hyperparameter Optimization for Colorectal Cancer Classification,” J. Electr. Syst., vol. 20, no. 3s, pp. 1–11, 2024.
O. Attallah, “Lung and Colon Cancer Classification Using Multiscale Deep Features Integration of Compact Convolutional Neural Networks and Feature Selection,” Technologies, vol. 13, no. 2, pp. 1–28, 2025, doi: 10.3390/technologies13020054.
B. Zhou, C. Han, and T. Guo, “Convergence of stochastic gradient descent in deep neural network,” Acta Math. Appl. Sin. English Ser., vol. 37, no. 1, pp. 126–136, 2021.
F. Huang, “Faster adaptive momentum-based federated methods for distributed composition optimization,” arXiv Prepr. arXiv2211.01883, 2022.
R. Ando and Y. Takefuji, “A Randomized Hyperparameter Tuning of Adaptive Moment Estimation Optimizer of Binary Tree-Structured LSTM,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 7, 2021.
R. Ando and Y. Takefuji, “A Randomized Hyperparameter Tuning of Adaptive Moment Estimation Optimizer of Binary Tree-Structured LSTM,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 7, 2021.
J. Liu, B. Li, Y. Zhou, X. Zhao, J. Zhu, and M. Zhang, “Online learning for dnn training: a stochastic block adaptive gradient algorithm,” Comput. Intell. Neurosci., vol. 2022, no. 1, p. 9337209, 2022.
B. H. Ekayanti, S. Prayogi, and S. Gummah, “Efforts to Drill the Critical Thinking Skills on Momentum and Impulse Phenomena Using Discovery Learning Model,” Int. J. Essent. Competencies Educ., vol. 1, no. 2, pp. 84–94, 2022.
Q. Yin, C. Han, A. Li, X. Liu, and Y. Liu, “A Review of Research on Building Energy Consumption Prediction Models Based on Artificial Neural Networks,” Sustainability, vol. 16, no. 17, p. 7805, 2024.
M. Madhiarasan and M. Louzazni, “Analysis of artificial neural network: architecture, types, and forecasting applications,” J. Electr. Comput. Eng., vol. 2022, no. 1, p. 5416722, 2022.
C. Beck, A. Jentzen, and B. Kuckuck, “Full error analysis for the training of deep neural networks,” Infin. Dimens. Anal. Quantum Probab. Relat. Top., vol. 25, Apr. 2022, doi: 10.1142/S021902572150020X.
K. L. Du, R. Zhang, B. Jiang, J. Zeng, and J. Lu, “Understanding Machine Learning Principles: Learning, Inference, Generalization, and Computational Learning Theory,” Mathematics, vol. 13, no. 3, pp. 1–58, 2025, doi: 10.3390/math13030451.
A. Ampavathi and V. Saradhib, “Multi disease-prediction framework using hybrid deep learning: an optimal prediction model,” Comput. Methods Biomech. Biomed. Engin., vol. 24, no. 10, pp. 1146–1168, 2021, doi: https://doi.org/10.1080/10255842.2020.1869726.
A. Ounajim et al., “Machine learning algorithms provide greater prediction of response to SCS than lead screening trial: a predictive AI-based multicenter study,” J. Clin. Med., vol. 10, no. 20, p. 4764, 2021.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Syaharuddin Syaharuddin, Abdillah Abdillah, Mariono Mariono, Saba Mehmood
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
-a.jpg)
-b.jpg)
