Visual speech recognition using attention-enhanced ResNet and hybrid recurrent-transformer encoder with quantity-ratio curriculum learning for low-resource languages

Abstrak

People with hearing impairments use visual cues through lipreading to get a better understanding of their communication. While many studies focus on lipreading research for English and Mandarin, there are still limited studies that address lipreading research for low-resource languages. Therefore, we establish a novel framework based on deep learning to solve the problem, which consists of several key components. First, it includes an optimized preprocessing stage and additional data augmentation for the lipreading task. Second, it incorporates a Spatial-Temporal Feature Extractor (STFE) to extract better lip movement features. Our STFE uses a channel and spatial attention module incorporated with ResNet-18 as an attention-enhanced ResNet and a combination of BiGRU with a multi-stage residual transformer encoder. Lastly, the model integrates a word classification module to make the final prediction. Moreover, we train the model using the quantity-ratio curriculum learning strategy by starting with easier categories and incrementally increasing difficulty. It maintains constant exposure to easier samples while gradually introducing more difficult ones until reaching the final stage with a defined quantity. In addition, we also present an expanded lipreading dataset for the Indonesian language (ID-LR), which is one of the low-resource languages. ID-LR contains more than 105,000 videos with 230-word classes spoken by diverse persons in natural conditions. With this configuration and learning strategy, our approach reaches remarkable performance in terms of accuracy and becomes state-of-the-art in multiple low-resource languages of lipreading datasets (Turkish, Romanian, German, and Indonesian) based on our presented benchmarks.

Identifier

• OpenAlex: https://openalex.org/W4413406819
• DOI: https://doi.org/10.1016/j.neucom.2025.131303