From Visual Perception to Context-Aware Instructions: Integrating Object Detection and LLMs for Navigation Assistance

Abstrak

Modern navigation systems often stop at visual perception, providing raw detections without translating them into actionable guidance for drivers. We present a practical, deployable navigation assistance framework that tightly couples real-time object detection with Large Language Models (LLMs) to produce fluent, context-aware driving instructions. Unlike conventional Advanced Driver Assistance Systems (ADAS), our approach introduces a structured intermediate representation that encodes detector outputs, such as traffic sign identities and locations, into a compact, machine-readable format before prompting the LLM. This design improves controllability, reduces irrelevant generation, and enables adaptation to dynamic roadway and traffic contexts. Evaluated on the 21-class Traffic Sign in Indonesia Dataset, our system achieves state-of-the-art detection performance and introduces the Feasibility Score, a multi-criteria human evaluation metric that captures relevance, coherence, completeness, fluency, and specificity of generated instructions. Experiments across multiple LLM configurations demonstrate that coupling structured perception with LLM-based reasoning produces guidance rated as clearer, more specific, and more context-relevant than perception-only or naive LLM baselines. These results position our framework as a concrete step toward next-generation, human-centered navigation assistance that bridges the gap between visual recognition and actionable driver communication.

Identifier

• OpenAlex: https://openalex.org/W7125441032
• DOI: https://doi.org/10.1145/3786554.3786576