Adaptive Multimodal Fusion Architecture for Heterogeneous Embedding Spaces in Retrieval-Augmented Generation Systems

Abstrak

Retrieval-augmented generation systems integrate external information to mitigate hallucinations in large language models, yet existing multimodal retrieval-augmented generation implementations struggle with heterogeneous embedding spaces from diverse sources. This study introduces an adaptive multimodal fusion architecture that addresses the critical challenge of integrating embeddings from research models and commercial application programming interfaces with varying dimensionalities (ranging from 768 to 1,536 dimensions) and training paradigms. We propose a source-agnostic framework comprising lightweight adapters, cross-modal attention mechanisms, and learned confidence calibration. The system was evaluated on 20,000 multimodal samples across image, video, audio, and text modalities, demonstrating significant performance improvements. The architecture enables end-to-end training without separate pretraining steps while maintaining flexibility for pretrained encoders. Experimental results show 7.2% improvement in Recall@10 compared to single-best source approaches (82.4% versus 89.6%), with expected calibration error of 0.048, indicating well-calibrated confidence scores. The framework achieves graceful degradation with only 3.8% performance reduction when one modality becomes unavailable and maintains 81.3% of in-domain performance in zero-shot domain transfer tasks. This work advances multimodal retrieval-augmented generation systems by enabling plug-and-play integration of heterogeneous embedding sources while preserving computational efficiency and interpretability.

Identifier

• DOI: https://doi.org/10.1109/access.2026.3675234
• OpenAlex: https://openalex.org/W7139003787