New Benchmark Evaluates Indirect Prompt Injection Risks in Retrieval-Augmented Generation Systems

Global: New Benchmark Evaluates Indirect Prompt Injection Risks in Retrieval-Augmented Generation Systems

On January 16, 2026, researchers Haoze Guo and Ziqi Wei submitted a paper to arXiv introducing OpenRAG‑Soc, a benchmark designed to assess indirect prompt injection and retrieval‑poisoning attacks on retrieval‑augmented generation (RAG) systems. The work aims to provide practitioners with a reproducible, realistic testing suite that measures how web‑native content can be manipulated during the ingestion and generation phases of RAG pipelines.

Background on Retrieval‑Augmented Generation

RAG architectures combine large language models with external knowledge sources, typically by retrieving user‑generated web content and feeding it into the generation step. This approach improves factual grounding but also expands the attack surface, as malicious actors can embed harmful instructions or misinformation within the retrieved documents.

Nature of Indirect Prompt Injection

Indirect prompt injection refers to adversarial content that survives preprocessing and influences the language model’s behavior without being directly presented as a prompt. When coupled with retrieval poisoning—where the indexed corpus is deliberately corrupted—the threat can lead to unintended or harmful model outputs.

OpenRAG‑Soc Benchmark Overview

OpenRAG‑Soc packages a curated social‑media corpus with interchangeable sparse and dense retrievers. It also incorporates three mitigations: HTML/Markdown sanitization, Unicode normalization, and attribution‑gated answering. The benchmark standardizes end‑to‑end evaluation, from data ingestion through response generation, enabling apples‑to‑apples comparisons across different carriers and defenses.

Evaluation Methodology

The suite records several metrics, including the time required for an attack to manifest at answer time, rank shifts observed in both sparse and dense retrieval stages, overall utility of the system, and latency overhead introduced by defenses. By reporting these figures, researchers can quantify trade‑offs between security and performance.

Implications for Practitioners

OpenRAG‑Soc targets developers and security teams responsible for deploying web‑facing RAG applications. Its modular design allows rapid integration into existing pipelines, facilitating continuous risk monitoring and the hardening of deployments against emerging injection techniques.

Future Directions

The authors suggest extending the benchmark to cover additional web formats, exploring adaptive mitigation strategies, and collaborating with industry partners to validate real‑world effectiveness. Continued community contributions are encouraged to keep the dataset and evaluation tools up to date.

This report is based on information from arXiv, licensed under Academic Preprint / Open Access. Based on the abstract of the research paper. Full text available via ArXiv.