COMPARATIVE ANALYSIS OF COGNITIVE DIAGNOSTIC MODELS IN POMDP-BASED ADAPTIVE TESTING
DOI:
https://doi.org/10.33480/jitk.v11i4.7601Keywords:
Computerized Adaptive Testing, Cognitive Diagnostic Model, KaNCD, MCD, POMDPAbstract
Recent progress in learning analytics and educational data mining has accelerated the development of adaptive learning systems, where Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) has emerged as a significant approach. CD-CAT employs Cognitive Diagnosis Models (CDMs) to generate detailed evaluations of student competencies. Nevertheless, increasing numbers of attributes and test items introduce challenges related to the complexity and uncertainty of adaptive policies. This research investigates and compares four cognitive diagnosis models, namely GD-DINA, MIRT, MCD, and KaNCD, within an adaptive testing framework based on a Partially Observable Markov Decision Process (POMDP). The evaluation was conducted using the ASSISTments dataset with Accuracy, AUC, and expected reward as performance metrics. The findings indicate that KaNCD achieved the best overall performance, obtaining the highest diagnostic accuracy (0.7503) and AUC score (0.7410), while also maintaining stable results in POMDP-based adaptive testing (expected reward = 0.792). Although GD-DINA produced the highest expected reward (0.901), its accuracy was comparatively lower. Meanwhile, MCD demonstrated a balanced performance, with high accuracy (0.7464) and strong adaptability of policy (reward = 0.873). Overall, these results suggest that KaNCD offers the most effective balance among accuracy, interpretability, and efficiency in POMDP-based adaptive testing systems.
Downloads
References
[1] S. B. H. Kate Churruca, Chiara Pomare, Louise A. Ellis, Janet C. Long, “Health Expectations - 2021 - Churruca - Patient‐reported outcome measures PROMs A review of generic and.pdf.” Sydney, NSW, Australia, 2021. doi: 10.1111/hex.13254.
[2] Y. Al Fadillah, A. R. Akbar, and Gusmaneli, “Strategi Desain Pembelajaran Adaptif untuk Meningkatkan Pengalaman Belajar di Era Digital,” J. Pendidik. Sains dan Teknol. Terap., vol. 01, no. 04, pp. 354–362, 2024.
[3] Y. Zhuang, Q. Liu, H. Bi, Z. Huang, W. Huang, J. Li, J. Yu, Z. Liu, Z. Hu, Y. Hong, Z. A. Pardos, H. Ma, M. Zhu, S. Wang, and E. Chen, “Survey of Computerized Adaptive Testing: A Machine Learning Perspective,” arXiv preprint arXiv:2404.00712, 2024. doi: 10.48550/arXiv.2404.00712.
[4] H. Luo, D. Wang, Z. Guo, Y. Cai, and D. Tu, “Combining Cognitive Diagnostic Computerized Adaptive Testing With Multidimensional Item Response Theory,” Appl. Psychol. Meas., vol. 46, no. 4, pp. 288–302, 2022, doi: 10.1177/01466216221084214.
[5] X. Gao, W. Ma, D. Wang, Y. Cai, and D. Tu, “A Class of Cognitive Diagnosis Models for Polytomous Data,” J. Educ. Behav. Stat., vol. 46, no. 3, pp. 297–322, 2021, doi: 10.3102/1076998620951986.
[6] X. Cao, Y. Lin, D. Liu, F. Zheng, and H. B. L. Duh, “Novel item selection strategies for cognitive diagnostic computerized adaptive testing: A heuristic search framework,” Behav. Res. Methods, vol. 56, no. 4, pp. 2859–2885, 2024, doi: 10.3758/s13428-023-02228-9.
[7] T. Xin, C. Wang, P. Chen, and Y. Liu, “Editorial: Cognitive Diagnostic Models: Methods for Practical Applications,” Front. Psychol., vol. 13, no. April, pp. 1–3, 2022, doi: 10.3389/fpsyg.2022.895399.
[8] Y. Lin et al., “A Survey on Reinforcement Learning for Recommender Systems,” IEEE Trans. Neural Networks Learn. Syst., vol. 35, no. 10, pp. 13164–13184, 2024, doi: 10.1109/TNNLS.2023.3280161.
[9] Y. S. Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, “NeuralCD: A General Framework for Cognitive Diagnosis,” IEEE Trans. Knowl. Data Eng., vol. 35, no. 8, pp. 8312–8327, 2023, doi: 10.1109/TKDE.2022.3201037.
[10] X. Xiang, S. Foo, and H. Zang, “Recent Advances in Deep Reinforcement Learning Applications for Solving Partially Observable Markov Decision Processes (POMDP) Problems Part 2—Applications in Transportation, Industries, Communications and Networking and More Topics,” Mach. Learn. Knowl. Extr., vol. 3, no. 4, pp. 863–878, 2021, doi: 10.3390/make3040043.
[11] A. K. Shakya, G. Pillai, and S. Chakrabarty, “Reinforcement learning algorithms: A brief survey,” Expert Syst. Appl., vol. 231, no. April, p. 120495, 2023, doi: 10.1016/j.eswa.2023.120495.
[12] S. Yang, L. Qin, and X. Yu, “Endowing Interpretability for Neural Cognitive Diagnosis by Efficient Kolmogorov-Arnold Networks,” arXiv preprint arXiv:2405.14399, 2024. doi: 10.48550/arXiv.2405.14399
[13] Z. Li, “Kolmogorov-Arnold Networks are Radial Basis Function Networks,” arXiv preprint arXiv:2405.06721, 2024. doi: 10.48550/arXiv.2405.06721.
[14] J. Jepkoech, D. M. Mugo, B. K. Kenduiywo, and E. C. Too, “The Effect of Adaptive Learning Rate on the Accuracy of Neural Networks,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 8, pp. 736–751, 2021, doi: 10.14569/IJACSA.2021.0120885.
[15] J. Schmidt-Hieber, “The Kolmogorov–Arnold representation theorem revisited,” Neural Networks, vol. 137, pp. 119–126, 2021, doi: 10.1016/j.neunet.2021.01.020.
[16] H. Ma, Y. Zeng, S. Yang, C. Qin, X. Zhang, and L. Zhang, “A novel computerized adaptive testing framework with decoupled learning selector,” Complex Intell. Syst., vol. 9, no. 5, pp. 5555–5566, 2023, doi: 10.1007/s40747-023-01019-1.
[17] Saad Hikmat Haji and Adnan Mohsin Abdulazeez, “Comparison Of Optimization Techniques Based On Gradient Descent Algorithm: A Review,” PalArch’s J. Archaeol. Egypt / Egyptol., vol. 18, no. 4 SE-, pp. 2715–2743, 2021, [Online]. Available: https://archives.palarch.nl/index.php/jae/article/view/6705
[18] J. Li, F. Wang, and J. Liu, “Towards the Identifiability and Explainability for Personalized Learner Modeling : An Inductive Paradigm,” 2024, doi: 10.1145/3589334.3645437.
[19] S. Dai, “Dealing with Missing Responses in Cognitive Diagnostic Modeling,” pp. 318–342, 2022.
[20] C. Ma, J. Ouyang, and G. Xu, “Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models" Psychometrika. vol 88. no. 1, pp. 175-207, 2023. doi:10.1007/s11336-022-09867-5”.
[21] C. Li, C. Ma, and G. Xu, “Learning Large Q-Matrix by Restricted Boltzmann Machines,” Psychometrika, vol. 87, no. 3, pp. 1010–1041, 2022, doi: 10.1007/s11336-021-09828-4.
[22] P. N. Srinivasu, N. Sandhya, R. H. Jhaveri, and R. Raut, “From Blackbox to Explainable AI in Healthcare: Existing Tools and Case Studies,” Mob. Inf. Syst., vol. 2022, 2022, doi: 10.1155/2022/8167821.
[23] X. Li, H. Xu, J. Zhang, and H. Chang, “Deep Reinforcement Learning for Adaptive Learning Systems,” vol. 48, no. 2, pp. 220–243, 2023, doi: 10.3102/10769986221129847.
[24] T. Yang and Y. Ying, “AUC Maximization in the Era of Big Data and AI: A Survey,” ACM Comput. Surv., vol. 55, no. 8, 2023, doi: 10.1145/3554729.
[25] J. Li, L. Ma, P. Zeng, and C. Kang, “New Item Selection Method Accommodating Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing : Maximum Deviation and Maximum Limitation Global Discrimination Indexes,” vol. 12, no. May, pp. 1–11, 2021, doi: 10.3389/fpsyg.2021.619771.
[26] O. Hospodarskyy, V. Martsenyuk, N. Kukharska, A. Hospodarskyy, and S. Sverstiuk, “Understanding the Adam Optimization Algorithm in Machine Learning,” CEUR Workshop Proc., vol. 3742, pp. 235–248, 2024.
[27] S. Ketabi, S. M. Alavi, and H. Ravand, “Diagnostic test construction: Insights from cognitive diagnostic modeling,” Int. J. Lang. Test., vol. 11, no. 1, pp. 22–35, 2021.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Mohamad Ridho Mubarok, Fandy Setyo Utomo
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
-a.jpg)
-b.jpg)
