Beyond mAP: A Critical Assessment of Reliability Evaluation Practices in Object Detection

Abstrak

Mean Average Precision (mAP) evaluated on static, independent and identically distributed benchmarks has become the de facto standard for assessing object detector quality. This paper argues that treating mAP as a sufficient proxy for deployment reliability creates a systematic blind spot with direct consequences for safety-critical applications. Through a critical review of the object detection evaluation literature, we identify three reliability dimensions that mAP does not measure and that existing evaluation pipelines do not address. First, confidence calibration is neglected and mischaracterized: reviewed detection calibration works uniformly evaluate large-capacity architectures and characterize miscalibration as overconfidence, leaving lightweight detectors under imbalanced training conditions unexamined. Second, intra-object confidence stability is formally undefined: the community has observed and engineered around frame-to-frame confidence fluctuation without proposing a standardized metric. We introduce the intra-track Coefficient of Variation and flickering rate as a first formalization of this dimension, operable without temporal ground-truth annotations. Third, distributional robustness is acknowledged as critical but structurally absent from evaluation pipelines. We further propose a four-dimensional evaluation framework integrating these gaps alongside detection accuracy and demonstrate, through observations across 12 configurations of three YOLO nano architectures, that simultaneous multi-dimensional evaluation surfaces reliability patterns that mAP-only assessment conceals.

Identifier

• DOI: https://doi.org/10.5281/zenodo.20582670
• OpenAlex: https://openalex.org/W7163808616