A Multimodal Transformer-LSTM Framework for Cross-Lingual Lexical Alignment of Indonesian Regional Languages
Abstrak
Research on cross-linguistic lexical alignment in Indonesian regional languages is still limited, especially in the use of multimodal data and the systematic evaluation of multilingual Transformer models. Most previous studies have relied on single-text data and semantic similarity-based approaches without providing adequate empirical evidence regarding the performance of literal lexical alignment derived from multimodal transcription. These limitations indicate the urgency for research to examine multimodal approaches in the task of cross-linguistic lexical alignment in Indonesian regional languages. This study contributes through the development and evaluation of a multimodal framework based on a Transformer and Long Short-Term Memory (LSTM) architecture for lexical alignment across languages. The novelty of this research lies in the utilization of primary field data from native speakers curated into a multimodal corpus of Indonesian regional languages, as well as in the comparative evaluation of a multilingual Transformer model for literal lexical alignment based on audio and visual transcription. Audio and visual data were transcribed using Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR) to produce cross-modality text representations. The representations were processed using the mBERT, DistilBERT, XLM-R Large, and LaBSE models, with the support of sequential modeling using LSTM. This dataset consists of 6,000 parallel multimodal utterances. Experimental results showed that the multilingual Transformer model was effective in performing cross-linguistic lexical alignment, with LaBSE producing the most consistent performance in terms of Accuracy, Mean Reciprocal Rank (MRR), and BLEU metrics. These findings make a significant empirical contribution to the study of multimodal-based cross-language lexical alignment for Indonesian regional languages.
Identifier
- OpenAlex: https://openalex.org/W7150182569
- DOI: https://doi.org/10.48084/etasr.17086