Emoji-aware multimodal representations for multi-label emotion analysis in Indonesian social media
Abstrak
Text Emotion Analysis (TEA) has progressed rapidly, but many studies still approach emotion as a single-label problem and focus solely on text. Research on TEA in low-resource languages such as Indonesian remains relatively scarce, and most available datasets do not explicitly capture paralinguistic cues, such as emojis, even though emojis often convey strong emotional signals. To bridge this gap, we propose an emoji-aware multimodal framework for multi-label emotion classification in Indonesian social media, using the eight Plutchik emotions as labels. We constructed the dataset using a semi-automated annotation pipeline: distant supervision was used to bootstrap labels, and experts then refined them by considering both text and emoji signals. The release contains 139,414 cleaned Indonesian social media posts from YouTube, X (formerly Twitter), Reddit, and Steam. We provide (i) a mixed corpus with posts both with and without emojis and (ii) an emoji-rich subset for emoji-aware experiments, along with the annotation guidelines and fixed train/validation/test splits. For all model variants, Indonesian Bidirectional Encoder Representations from Transformers (IndoBERT) serves as the text backbone; the only difference lies in how emoji information is incorporated. We tested three integration options for emojis: adding them early (by concatenation or tokenizer expansion), fusing them mid-stream with cross-attention, or combining predictions late (average/weighted). Compared to a text-only baseline, emoji-aware models perform better overall, reflecting the fact that emojis often convey emotional cues that text alone misses. The best scores come from tokenizer expansion (micro-averaged F1 = 0.8554; macro-averaged F1 = 0.7769), likely because emojis are processed alongside words through IndoBERT’s contextual layers. Beyond model comparisons, we also introduce an Indonesian emoji-aware multi-label dataset and release fixed splits and guidelines to support reproducible research.
Identifier
- OpenAlex: https://openalex.org/W7162462184
- DOI: https://doi.org/10.7717/peerj-cs.3927