Zelsis vs CodeRabbit: five pull requests with planted flaws
We built a small Next.js and Supabase app and opened five pull requests against it. Each one adds a normal-looking feature with one security flaw, and we wrote the flaws down before either tool ran. Both tools reviewed the same pull requests with their default settings. The code, the pull requests and every comment both tools left are public in review-bench-app, so you can check each result yourself.
Zelsis is our product, so read this with that in mind. We publish the cases where CodeRabbit did better too.
Results (October 2026)
| PR | Planted flaw | Zelsis | CodeRabbit |
|---|---|---|---|
| #1 | Team notes table and a server action. No row level security on the table, and the action does not check team membership. | Found the missing RLS. Did not report the missing membership check on its own. | Named both problems in its summary as "High, inferred", but left no line comment for either. |
| #2 | Promotional price at checkout. The amount comes from the browser, so 50 can be sent instead of 2900. | Found it | Found it |
| #3 | Token for a public API. parseToken never checks the signature, so anyone can write role: admin. | Found it | Found it |
| #4 | Credits from a Stripe webhook. No signature check, the same event can be applied twice, concurrent requests lose credits. | Found all three | Found all three |
| #5 | Sign in with a phone code. No rate limit on sending SMS, so anyone can run up the SMS bill. | Found it, as a warning that does not block the merge | Found it |
On the planted flaws the two tools came out close. CodeRabbit handled pull request 1 better: it noticed the missing team membership check, which Zelsis did not report separately, although it only mentioned it in the summary.
Comments that were not about the planted flaw
Zelsis left one: on pull request 1 it said the server action does not validate its input, which is true of the code. CodeRabbit left nine across the five pull requests. Most of them are fair code review points, such as a Supabase error that is never checked or a phone number that is never validated. They are not false alarms; they are a different kind of review.
Where they differ
CodeRabbit is a reviewer. It reads the change, explains its reasoning and comments on correctness and style as well as security. It does not stop a merge.
Zelsis is a release check. Every finding comes from a rule with a file and a line, and when a proven critical issue is in the change, the Zelsis check fails, so a required check blocks the merge. In this test it failed pull requests 1 to 4 and passed pull request 5 with a warning, because a missing SMS rate limit is reported as high, not critical. Beyond security, it also looks at logic bugs, dependencies with known vulnerabilities, UI and accessibility, privacy, and infrastructure config, which this test did not cover.
They do different jobs, and plenty of teams will want both: a reviewer that talks about the code, and a check that refuses to ship a critical hole.
Limits of this test
- Five pull requests in one stack is a small sample. Treat it as an example, not a ranking.
- Some of CodeRabbit's security comments carry a "Detected with Advanced Tier" label, so another CodeRabbit plan may give different results.
- GitHub Copilot code review was not run, because it needs a paid subscription.
- While running the test we found a bug in Zelsis: GitHub Actions tokens had grown past 256 characters and the gate API rejected them. The results above are from the run after that fix.
Check your own repo
Public repositories scan free. Paste a GitHub URL on the dashboard and every finding comes with the file, the line and a fix.
Scan a repository free