When Retrieval Hurts: An Honest Evaluation of RAG for Solidity Vulnerability Detection

Abstract

Empirical study showing a sample-size sign reversal in naive RAG for Solidity vulnerability detection: +2.0% Macro-F1 at n=100 flips to -2.7% at n=250 on SolidiFI. Argues for bootstrap confidence intervals in any RAG evaluation.

Key questions

Q. Does RAG improve LLM detection of Solidity smart-contract vulnerabilities?
Not reliably. On the SolidiFI benchmark, naive RAG improved Macro-F1 by +2.0% at n=100, but the gain reversed to −2.7% at n=250 — a sample-size sign reversal within noise. The paper argues any RAG evaluation must report bootstrap confidence intervals before claiming an improvement.

Q. Why can a measured RAG improvement disappear with more test data?
A small test set can show an apparent gain that shrinks or reverses as the sample grows; without confidence intervals it is indistinguishable from noise. This study demonstrates the effect directly on Solidity vulnerability detection.