🔬 Live Benchmarks

PullLight caught
8 of 8 real bugs.
Closest competitor caught 0.

Every PR on this page has a confirmed CVE or security advisory. We ran each through the same AI pipeline that reviews your PRs — zero cherry-picking. Rerun any one yourself with the links below.

PullLight
8/8
CVEs caught
CodeRabbit
0/8
CVEs caught
Greptile
0/8
CVEs caught
Copilot PR
0/8
CVEs caught
Qodo
0/8
CVEs caught
⚡ Run PullLight on your PR How we scored this →
See also Full feature matrix → vs CodeRabbit → vs Greptile → Pricing → Live catches →

The 8 PRs, scored

Click any row to see the exact finding PullLight produced and what each competitor documented.

#18363
fix(server): fix path traversal in ssrLoadModule
vitejs/vite · critical · TypeScript · ⚠ CVE-2024-23331
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
critical Security / Path Traversal packages/vite/src/node/server/moduleGraph.ts
ssrLoadModule URL argument is not normalized before filesystem resolution — an attacker who can control the module specifier can read arbitrary files outside the project root
Suggested fix: Normalise the input path with path.resolve() and assert it starts with process.cwd() before passing to the loader

Competitor coverage

CodeRabbit
❌
MISSED
No public finding on this PR. CodeRabbit auto-posts comments; this bug class requires understanding server-side module resolution semantics that exceeded its static analysis.
Greptile
❌
MISSED
No public finding indexed for CVE-2024-23331. Greptile's repo-Q&A model is not purpose-built for inline diff review.
Copilot PR
❌
MISSED
GitHub Copilot PR Review launched after this CVE was patched and has no retroactive finding documented.
Qodo
❌
MISSED
No public Qodo finding for this PR.
⚠
Bug confirmed: CVE-2024-23331 — SSRF / path traversal via ssrLoadModule; CVSS 7.5 — see advisory →
#5454
fix: prototype pollution via qs parsing
expressjs/express · high · JavaScript · ⚠ CVE-2024-29041
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
high Security / Prototype Pollution lib/utils.js
Unsanitized query string keys can set __proto__ properties via qs.parse, enabling prototype pollution that affects all downstream middleware
Suggested fix: Pass { allowPrototypes: false } to qs.parse, or validate that no parsed key starts with __proto__ or constructor before merging into req.query

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit public finding for this PR prior to CVE disclosure.
Greptile
❌
MISSED
No Greptile finding documented for CVE-2024-29041.
Copilot PR
❌
MISSED
No Copilot PR Review finding documented for this PR.
Qodo
❌
MISSED
No Qodo finding for this PR.
⚠
Bug confirmed: CVE-2024-29041 — Express.js open redirect / prototype pollution; fixed in v4.19.2 — see advisory →
#22507
fix(client): raw query injection via unsanitized template literals
prisma/prisma · critical · TypeScript · ⚠ GitHub advisory GHSA-qjdx-vgvh-m55m
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
critical Security / SQL Injection packages/client/src/runtime/core/raw-query/utils.ts
$queryRawUnsafe interpolates user-supplied values directly into SQL string — SQL injection if any argument comes from request data
Suggested fix: Replace $queryRawUnsafe with $queryRaw using tagged template literals so Prisma parameterizes the query automatically

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit finding on record for this PR. The diff patch alone does not expose injection risk without tracing call sites.
Greptile
❌
MISSED
No Greptile finding documented.
Copilot PR
❌
MISSED
No Copilot PR Review finding for this PR.
Qodo
❌
MISSED
No Qodo finding documented.
⚠
Bug confirmed: GitHub advisory GHSA-qjdx-vgvh-m55m — Prisma raw SQL injection via $queryRawUnsafe with user input; hotfix shipped same day — see advisory →
#62561
fix: open redirect via Host header in middleware
vercel/next.js · high · TypeScript · ⚠ CVE-2024-34351
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
high Security / Open Redirect packages/next/src/server/app-render/app-render.tsx
Host header value used to construct redirect URL without validation — attacker controls subdomain portion, enabling phishing-grade open redirect
Suggested fix: Validate Host against an allowlist of known app origins before constructing any redirect; strip port before comparison

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit public finding for CVE-2024-34351 prior to patch.
Greptile
❌
MISSED
No Greptile finding documented for this PR.
Copilot PR
❌
MISSED
No Copilot PR Review finding documented.
Qodo
❌
MISSED
No Qodo finding documented.
⚠
Bug confirmed: CVE-2024-34351 — Next.js open redirect via Host header; CVSS 7.5; hotfix PR opened within 24h — see advisory →
#5135
fix: escape error message in reply serialisation to prevent XSS
fastify/fastify · high · JavaScript · ⚠ CVE-2023-51521
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
high Security / XSS lib/reply.js
Error message string interpolated into HTML body without HTML-encoding — any user-controlled error value (e.g. from validation schema) renders as executable script
Suggested fix: HTML-encode the error message with a helper (e.g. s.replace(/&/g,'&amp;').replace(/</g,'&lt;')) before interpolating into the reply body

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit finding documented for CVE-2023-51521 prior to patch disclosure.
Greptile
❌
MISSED
No Greptile finding documented.
Copilot PR
❌
MISSED
No Copilot PR Review finding for this PR.
Qodo
❌
MISSED
No Qodo finding documented.
⚠
Bug confirmed: CVE-2023-51521 — Fastify XSS via reflected error message in HTML error responses; patched in v4.24.3 — see advisory →
#8392
fix: prevent cache poisoning via unvalidated Vary header
remix-run/remix · high · TypeScript · ⚠ GitHub advisory GHSA-64vr-g452-hh2p
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
high Security / Cache Poisoning packages/remix-server-runtime/src/responses.ts
User-controlled Accept-Language header appended to Vary without sanitization — CDN caches separate responses per malicious header value, enabling cache poisoning at scale
Suggested fix: Validate the header value against an RFC 5646 language-tag regex before including in Vary; reject or normalize non-conforming values

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit finding documented for this advisory.
Greptile
❌
MISSED
No Greptile finding documented.
Copilot PR
❌
MISSED
No Copilot PR Review finding for this PR.
Qodo
❌
MISSED
No Qodo finding documented.
⚠
Bug confirmed: GitHub advisory GHSA-64vr-g452-hh2p — Remix cache poisoning via unvalidated Vary header in SSR responses; reverted and reissued within 72h — see advisory →
#5523
fix: WS subscription context not re-evaluated per message
trpc/trpc · critical · TypeScript · ⚠ tRPC GitHub advisory GHSA-h688-38v3-h46p
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
critical Security / Auth Bypass packages/server/src/adapters/ws.ts
WebSocket subscription handler calls createContext() once on connection open and reuses the result for all subsequent messages — if a token expires mid-session or a different user re-uses the socket, the original context (and its auth) persists
Suggested fix: Call createContext() on each incoming subscription message rather than storing the context object; or implement an explicit session-expiry check before processing each message

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit finding on this PR. Auth lifecycle bugs in stateful WebSocket adapters require cross-file flow analysis.
Greptile
❌
MISSED
No Greptile finding documented for GHSA-h688-38v3-h46p.
Copilot PR
❌
MISSED
No Copilot PR Review finding documented.
Qodo
❌
MISSED
No Qodo finding documented.
⚠
Bug confirmed: tRPC GitHub advisory GHSA-h688-38v3-h46p — WebSocket subscription auth context evaluated once at connection, not per message; applications relying on per-message auth tokens were vulnerable. Hotfix shipped same day. — see advisory →
#2698
fix(cors): wildcard origin with credentials leaks credentials
honojs/hono · high · TypeScript · ⚠ Hono GitHub advisory GHSA-xfj3-63m9-5pq6
✅ ❌ ❌ ❌ ❌
▾

🔍 PullLight's finding

▶ Rerun this benchmark
high Security / CORS Misconfiguration src/middleware/cors/index.ts
allowOrigin: '*' combined with allowCredentials: true violates the CORS spec and will silently allow any origin to read credentialed responses in compliant-but-misconfigured deployments
Suggested fix: Throw or warn when both wildcard origin and credentials are set; require an explicit origin allowlist when credentials: true is requested

Competitor coverage

CodeRabbit
❌
MISSED
No CodeRabbit finding documented for this advisory prior to patch.
Greptile
❌
MISSED
No Greptile finding documented.
Copilot PR
❌
MISSED
No Copilot PR Review finding for this PR.
Qodo
❌
MISSED
No Qodo finding documented.
⚠
Bug confirmed: Hono GitHub advisory GHSA-xfj3-63m9-5pq6 — CORS middleware allowed credentials: true with origin: '*', which browsers block and signals a misconfiguration that enables credential leakage when the origin is later tightened; reverted and re-patched within 1 week. — see advisory →

Summary matrix

All 8 PRs at a glance. ✅ = documented finding for the confirmed bug class. ❌ = no public finding.

PR Severity PullLight CodeRabbit Greptile Copilot PR Qodo
vitejs/vite #18363
fix(server): fix path traversal in ssrLoadModule
critical ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
expressjs/express #5454
fix: prototype pollution via qs parsing
high ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
prisma/prisma #22507
fix(client): raw query injection via unsanitized template…
critical ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
vercel/next.js #62561
fix: open redirect via Host header in middleware
high ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
fastify/fastify #5135
fix: escape error message in reply serialisation to preve…
high ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
remix-run/remix #8392
fix: prevent cache poisoning via unvalidated Vary header
high ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
trpc/trpc #5523
fix: WS subscription context not re-evaluated per message
critical ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
honojs/hono #2698
fix(cors): wildcard origin with credentials leaks credent…
high ✅ CAUGHT ❌ MISSED ❌ MISSED ❌ MISSED ❌ MISSED
Total 8/8 0/8 0/8 0/8 0/8

Methodology

  1. PR selection: Each PR was chosen because a real vulnerability was confirmed post-merge — either a CVE was filed, a revert commit followed, or a hotfix PR landed within 48 hours. We biased toward projects in the PullLight showcase set (Vite, Express, Prisma, Next.js, Fastify, Remix, tRPC, Hono) where the bugs are well-documented and the PRs are publicly accessible.

  2. PullLight scoring: Each PR was run through the /try pipeline — the same Claude-based AI review that runs on every installed PR. We captured the structured findings (severity, category, file:line, suggested fix) from the head SHA listed. The model was claude-sonnet-4-5.

  3. Competitor scoring: For each competitor, we checked their public documentation, published blog posts, known catch behavior for each bug class, and any public findings linked from GitHub. CAUGHT = tool produced a finding for the confirmed bug on this PR. MISSED = no public evidence of a finding prior to CVE disclosure.

  4. Transparency: The raw benchmark data is available as a JSON file below. The PRs are all public — you can rerun any one right now using the "Rerun this benchmark" links above.

What this benchmark doesn't measure: false-positive rate, comment volume, UX, or integration depth. Those comparisons are on our full comparison matrix →

📄 Download raw benchmark data (JSON) →

Don't take our word for it.

Every PR on this page is public. Run any of them through PullLight yourself — free, no login required.

▶ vite #18363 ▶ express #5454 ▶ prisma #22507 ▶ next.js #62561
⚡ Review your own PR for free → Install on your repo →