IMPLEMENTING RETRIEVAL-AUGMENTED GENERATION AND VECTOR DATABASES FOR CHATBOTS IN PUBLIC SERVICES AGENCIES CONTEXT
Abstract
Rapid developments in information technology, such as chatbots and generative artificial intelligence, have drastically lowered the cost of providing services to the society. This study aims to measure performance of developed chatbot using retrieval augmented generation and vector database. This research compares the performance of existing Large Language Modelling (LLM) in answering questions related to regulations concerning public service agencies.. Using a vector database, questions are assessed and answered by the LLM model, considering cosine similarity scores. The best-performing model, gpt-4, is selected for the deployment process which have average cosine similarity score 0,404. The use of LLM for chatbot creation at the prototyping stage can provide a good response to the question asked related to public service agencies with retrieval augmented generation (RAG) process through regulation-based document extraction.
Downloads
References
“Gartner Top 10 Strategic Technology Trends 2024,” 2024, [Online]. Available: https://www.gartner.com/en/articles/gartner-top-10-strategic-technology-trends-for-2024
L. Perri, “What’s New in the 2023 Gartner Hype Cycle for Emerging Technologies.” Accessed: Nov. 12, 2023. [Online]. Available: https://www.gartner.com/en/articles/what-s-new-in-the-2023-gartner-hype-cycle-for-emerging-technologies#:~:text=What’s New in the 2023 Gartner Hype Cycle for Emerging Technologies&text=They fit into four main,human-centric security and privacy.
R. Kasali, Disruption. Jakarta: Gramedia, 2017.
T. Chen, M. Gascó-Hernandez, and M. Esteve, “The Adoption and Implementation of Artificial Intelligence Chatbots in Public Organizations: Evidence from U.S. State Governments,” Am. Rev. Public Adm., vol. 54, no. 3, pp. 255–270, 2024, doi: 10.1177/02750740231200522.
R. Qasem, B. Tantour, and M. Maree, “Towards the Exploitation of LLM-based Chatbot for Providing Legal Support to Palestinian Cooperatives,” arXiv preprint arXiv:2306.05827, 2023, doi: https://doi.org/10.48550/arXiv.2306.05827
Z. W. Lim et al., “Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard,” eBioMedicine, vol. 95, p. 104770, 2023, doi: 10.1016/j.ebiom.2023.104770.
M. Maryamah, M. M. Irfani, E. B. Tri Raharjo, N. A. Rahmi, M. Ghani, and I. K. Raharjana, “Chatbots in Academia: A Retrieval-Augmented Generation Approach for Improved Efficient Information Access,” KST 2024 - 16th Int. Conf. Knowl. Smart Technol., pp. 259–264, 2024, doi: 10.1109/KST61284.2024.10499652.
U. Shukla, S. Singh, A. Pundir, and G. J. Saxena, “Large language model based framework for knowledgebase coverage and correctness using chatbot and human feedback,” 2023 IEEE 7th Conf. Inf. Commun. Technol. CICT 2023, pp. 1–7, 2023, doi: 10.1109/CICT59886.2023.10455408.
M. Dean, R. R. Bond, M. F. McTear, and M. D. Mulvenna, “ChatPapers: An AI Chatbot for Interacting with Academic Research,” 2023 31st Irish Conf. Artif. Intell. Cogn. Sci. AICS 2023, pp. 1–7, 2023, doi: 10.1109/AICS60730.2023.10470521.
B. Zhong, W. He, Z. Huang, P. E. D. Love, J. Tang, and H. Luo, “A building regulation question answering system: A deep learning methodology,” Adv. Eng. Informatics, vol. 46, no. October, p. 101195, 2020, doi: 10.1016/j.aei.2020.101195.
M. Keuangan, Peraturan Menteri Keuangan Nomor 129 tahun 2020 tentang Pedoman Pengelolaan Badan Layanan Umum. 2020.
Maziyank, “Indonesian Regulation Text Parser.” Accessed: Feb. 23, 2024. [Online]. Available: https://github.com/maziyank/anali%0Asa-regulasi
E. Akdemir and N. Barışçı, “A review on deep learning applications with semantics,” Expert Syst. Appl., vol. 251, no. December 2021, 2024, doi: 10.1016/j.eswa.2024.124029.
J. W. Rae et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher,” 2021, [Online]. Available: http://arxiv.org/abs/2112.11446
J. Ye et al., “A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models,” pp. 1–47, 2023, [Online]. Available: http://arxiv.org/abs/2303.10420
OpenAI et al., “GPT-4 Technical Report,” vol. 4, pp. 1–100, 2023, [Online]. Available: http://arxiv.org/abs/2303.08774
R. Anantha, T. Bethi, D. Vodianik, and S. Chappidi, “Context Tuning for Retrieval Augmented Generation,” UncertaiNLP 2024 - Work. Uncertainty-Aware NLP, Proc. Work., pp. 15–22, 2024, doi: https://doi.org/10.48550/arXiv.2312.05708
P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” Adv. Neural Inf. Process. Syst., vol. 33, 2020-Decem, 2020.
S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Rana, and S. Nanayakkara, “Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering,” Trans. Assoc. Comput. Linguist., vol. 11, pp. 1–17, 2023, doi: 10.1162/tacl_a_00530.
J. Tekli, G. Tekli, and R. Chbeir, Combining Offline and On-the-fly Disambiguation to Perform Semantic-aware XML Querying, vol. 29, no. 1. 2023. doi: 10.2298/CSIS220228063T.
D. U. Sinha and M. V. Dubey, “The Technique of Different Semantic Search Engines,” Int. J. Recent Technol. Eng., vol. 9, no. 1, pp. 1496–1501, 2020, doi: 10.35940/ijrte.a2249.059120.
D. Banerjee, P. Singh, A. Avadhanam, and S. Srivastava, “Benchmarking LLM powered Chatbots: Methods and Metrics,” 2023, [Online]. Available: http://arxiv.org/abs/2308.04624
L. G. Gunnell, B. Nicholson, and J. D. Hedengren, “Equation-based and data-driven modeling: Open-source software current state and future directions,” Comput. Chem. Eng., vol. 181, no. October 2023, p. 108521, 2024, doi: 10.1016/j.compchemeng.2023.108521.
OpenAI, “Text Generation Model.” Accessed: Nov. 11, 2023. [Online]. Available: https://platform.openai.com/docs/guides/tex%0At-generation
Copyright (c) 2024 Ibnu Pujiono, Irfan Murtadho Agtyaputra, Yova Ruldeviyani
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
-a.jpg)
-b.jpg)
