Implementation of Fine-Tuning LLM Model on New Student Registration Chatbot at Muhammadiyah University of Makassar

Abstrak

This study implements fine-tuning of the Large Language Model (LLM) Gemini 2.5 Flash via Vertex AI to develop an information service chatbot for New Student Admissions (PMB) at Universitas Muhammadiyah Makassar. The process involved collecting 1,430 question–answer data pairs, preprocessing, augmentation, training the model for 37 epochs, and integrating it with Flask. The evaluation was conducted using BLEU, ROUGE-L, and the Customer Satisfaction Score (CSAT). The results indicate an accuracy improvement of 0.976, a reduction in loss from 2.0 to 0.07, a ROUGE-L recall score of 0.534, a BLEU score of 0.235, and a user satisfaction rate of 87.5%. These findings suggest that fine-tuning LLMs enhances semantic relevance and user satisfaction compared to rule-based NLP methods, Naïve Bayes, FastText-LSTM, and AIML. Despite limitations such as bias and high computational demands, this chatbot proves effective in the digitalization of higher education services.

Identifier

• DOI: https://doi.org/10.31763/iota.v5i3.1010
• OpenAlex: https://openalex.org/W4415946128