Comparative Evaluation of Rule-Based and Large Language Models for Financial Transaction Extraction in Chatbots

Abstrak

The increasing use of digital financial services requires tools that allow users to record and manage personal financial transactions efficiently. However, many conventional applications still rely on form-based interfaces that may reduce user engagement in daily financial recording. This study evaluates a conversational personal financial recording system that uses natural language processing to convert informal user messages into structured transaction data. The system was developed using the prototyping method and implemented through a messaging-based interface, a backend service, and a natural language processing module. The evaluation used a dataset of 300 conversational financial messages annotated with intent, amount, and category. The study compares a rule-based baseline with two open-source large language models, using intent accuracy, entity extraction metrics, output validity, user acceptance testing, and the system usability scale. The results show that open-source large language models achieved the best performance across the natural language processing evaluation, with strong intent classification, high entity extraction quality, and complete output validity. The user acceptance testing involving 30 participants produced an average success rate of 97.3%, while the system usability scale score reached 82.5, indicating excellent usability. These findings suggest that prompt-constrained large language models can improve conversational financial recording by providing reliable structured extraction and an accessible user experience.

Identifier

• DOI: https://doi.org/10.33395/sinkron.v10i2.16020
• OpenAlex: https://openalex.org/W7160443664