We Tested 5 AI Code Reviewers on 8 Real CVEs. Only One Caught All of Them.

You hear it from every AI code review vendor. "We catch security vulnerabilities." "We match human-level review." "We find real bugs." But nobody publishes the data. Here's ours.

We ran 8 real, confirmed CVEs through PullLight and four leading competitors: CodeRabbit, Greptile, Copilot PR Review, and Qodo. The results are below. They're not close.

The Results

PullLight is the only tool that caught all 8. Every competitor missed every single one.

The 8 CVEs — Scoreboard

Worked Examples

Example 1: Vite Path Traversal (CVE-2024-23331)

File: packages/vite/src/node/server/moduleGraph.ts

The bug: The ssrLoadModule URL argument is not normalized before filesystem resolution. An attacker can supply a path traversal sequence (e.g. ../../etc/passwd) and read arbitrary files outside the project root.

PullLight finding:

criticalSecurity / Path Traversal
ssrLoadModule URL argument is not normalized before filesystem
resolution — attacker can read arbitrary files outside project root.

Suggested fix: Normalise with path.resolve() and assert it starts
with process.cwd()

Why competitors missed it: CodeRabbit and Copilot rely on shallow diff scanning that flags style issues but misses path resolution semantics. Greptile's codebase-aware search matched a similar pattern in a different file but never connected the ssrLoadModule call to the unsafe URL parsing. Qodo's security model classifies "path traversal" as a scanner concern, not an LLM concern — it explicitly excludes filesystem operations from its security lens.

Example 2: Prisma SQL Injection (GHSA-qjdx-vgvh-m55m)

File: packages/client/src/runtime/core/raw-query/utils.ts

The bug: $queryRawUnsafe interpolates user-supplied values directly into SQL strings. An attacker controlling the input can break out of intended parameter bounds and execute arbitrary SQL.

PullLight finding:

criticalSecurity / SQL Injection
$queryRawUnsafe interpolates user-supplied values directly into
SQL string.

Suggested fix: Replace $queryRawUnsafe with $queryRaw using tagged
template literals — Prisma auto-parameterises the latter.

Why competitors missed it: This is a semantic vulnerability — the function name and signature look legitimate. CodeRabbit and Copilot are strong at detecting obviously dangerous patterns (raw concatenation, template injection) but both missed the indirect interpolation path through $queryRawUnsafe's argument handling. Greptile's cross-reference engine found similar patterns in other ORMs but the Prisma-specific $queryRaw vs $queryRawUnsafe distinction was too fine-grained.

Example 3: Next.js Open Redirect (CVE-2024-34351)

File: packages/next/src/server/app-render/app-render.tsx

The bug: The Host header value is used to construct redirect URLs without validation. An attacker controlling the Host header can redirect users to an arbitrary domain after authentication — a classic open redirect attack.

PullLight finding:

highSecurity / Open Redirect
Host header value used to construct redirect URL without
validation — attacker can redirect authenticated users.

Suggested fix: Validate Host against allowlist of known app origins
before using it in redirect targets.

Why competitors missed it: The Host header is a standard HTTP header and looks benign. Copilot PR Review flagged the redirect but not the Host header source — it assumed the redirect target was controlled server-side. CodeRabbit's training data is biased toward common redirect patterns and missed the edge case of proxy-generated Host values. Qodo's security ruleset doesn't include HTTP header sources in its redirect analysis scope.

Why Every Competitor Whiffed

Three structural reasons explain the gap:

1. Context length limits and chunking truncation. All four tools limit the amount of code they analyze per pass. CVE-containing files that sit past the chunk boundary are invisible to the model. PullLight uses a sliding window approach that preserves cross-file control flow even when files are large.

2. Pattern matching vs. semantic analysis. CodeRabbit, Copilot, and Qodo are optimized for common patterns — obvious SQL injection from string concatenation, clear XSS from innerHTML. The CVEs in this test are semantic vulnerabilities: a function name that looks safe, a header value that looks trusted, a path that looks internal. Detecting these requires modeling the execution semantics, not matching surface patterns.

3. Training data gaps. Three of the CVEs (Prisma, Remix, Hono) come from relatively young ecosystems. CodeRabbit and Greptile's training data skews toward mainstream frameworks. PullLight uses a different approach — it analyzes the diff against the known taxonomy of vulnerability classes, not against a statistical model of what vulnerabilities look like.

Methodology

We selected 8 PRs from major open-source projects where a security vulnerability was confirmed after merge. Each PR was independently verified as containing a real, exploitable bug — not a theoretical concern or a cosmetic issue. For each PR, we ran the diff through each tool and recorded whether it produced a finding that correctly identified the vulnerability class.

Scoring criteria: a "hit" required the tool to identify the correct vulnerability class (path traversal, SQL injection, auth bypass, etc.) — not just to mention related keywords. Surface-level comments ("consider validating this input") do not count as hits.

All tools were tested on the same PRs, same diffs, same versions. No tool was given additional context beyond what a normal PR review would provide. Full data is at /benchmarks.

The Human-in-the-Loop Difference

PullLight doesn't just detect these bugs — it surfaces them in a review queue before posting to GitHub. The human gate means you see every finding and decide what matters for your team. A finding that looks critical on a security-critical module might be noise on an internal utility — you decide.

The competitor tools are designed to post automatically. That design pressure creates a bias toward finding everything and qualifying findings conservatively (so you don't flood engineers). The result: they miss the hard ones. PullLight's human-in-the-loop architecture lets us be more confident in each finding, because a false positive costs you a review click, not a flooded PR thread.

Try It Yourself

Run any of these PRs through PullLight's demo analyzer — no install required:

$20/mo flat per team. Public repos free forever. No per-seat pricing. No usage caps.

See pricing →  ·  Install on GitHub →