TRANSFORMER-BASED GENERATIVE CHATBOT FOR HIGHER EDUCATION INFORMATION SERVICES WITH HYPERPARAMETER OPTIMIZATION AND MODEL EVALUATION

Abstract

Generative AI has revolutionized the creation of realistic multimedia content, including chatbots that generate human-like responses. This technology significantly improves higher education by enabling universities to provide fast, accurate, and efficient information services. Generative chatbots can handle multiple users simultaneously and operate 24/7, increasing productivity and accessibility. The model was developed using Machine Learning Lifecycle (MLLC) with deep learning algorithm and Transformer architecture. The dataset used consists of 5,403 question-answer pairs from Institut Teknologi Garut (ITG), which are divided into 5,089 pairs for training and 314 pairs for testing. From 12 hyperparameter configurations, the best combination (maxlen 80, num_layers 2, batch_size 128, embedding_dim 256, fully_connected_dim 256, num_heads 2, positional_encoding_length 512, learning_rate 0.0002, and epoch 100) achieved a BLEU score of 71.03% on the ITG dataset. Evaluation using ROUGE and METEOR also shows consistent performance, indicating good content coverage and semantic similarity. Retraining with another dataset using the same approach resulted in a slightly higher BLEU score of 72.05%, with a different optimal learning rate of 0.00025. The results of this study indicate that Transformer-based generative chatbots can support higher education services and highlight the importance of adjusting hyperparameters based on dataset characteristics. This research also provides opportunities to develop similar models in other universities by adapting datasets and exploring more advanced methods to improve performance in broader educational contexts.