Downscaling-Triggered Visual Prompt Injection: A Systematic Evaluation of Image-Scaling Attacks on Vision–Language Models

Abstrak

We study a new class of visual prompt injection (VPI) attacks that weaponize the image downscaling pipeline of modern vision–language models (VLMs). Building on the anamorpher tool, we generate anamorphic images that appear benign at full resolution but reveal a malicious text instruction only after the model’s standard resize-and-downscale operations. Under a threat model in which an adversary controls only the image while a benign user issues a harmless query, we ask: How often do such payloads survive preprocessing, and how often do they hijack the model's behavior? We evaluate three widely used open-source VLMs—llava-hf/llava-1.5-7b-hf, InternVL2.5-8B, and Qwen2-VL-7B. For each model, we fingerprint its effective preprocessing (resolution, library, and interpolation kernel), then construct a benchmark of 10,000 anamorphic attacks by combining 100 harmful payloads from JailBreakV-28K with 100 benign MS COCO images. We introduce metrics that separate (i) payload rendering (PRR), (ii) attack landing (ASR-VPI), and (iii) full safety bypass (ASR-C). Our results show that payloads can be reliably rendered after downscaling (PRR of 92.5%, 68.0%, and 55.7% for LLaVA-1.5, InternVL2.5, and Qwen2-VL, respectively), and that they frequently influence model behavior (ASR-VPI of 85.0%, 55.0%, and 40.0%). LLaVA-1.5 is particularly vulnerable, with 70.0% of attacks resulting in complete safety bypass. We conclude that image preprocessing is a critical but often overlooked attack surface for VLMs, and we highlight the need for future work on defenses that jointly harden the scaling pipeline and strengthen multimodal safety mechanisms while preserving benign utility.

Identifier

• OpenAlex: https://openalex.org/W7163739504
• DOI: https://doi.org/10.62527/joiv.10.3.4731