Rule-based vs AI code review: reproducible findings at merge time

Teams comparing rule-based vs AI code review are usually asking one question: can I trust this comment enough to block a merge? Fluent PR summaries from generic bots sound helpful until the same diff passes on Tuesday and fails on Thursday with no code change. Rule-based review answers with a rule id, a severity, and output you can re-run on your laptop.

This post contrasts the two approaches, when each helps, and how Critiq fits today: deterministic @critiq/rules checks locally and in GitHub Actions, not a hosted model reviewer. For the broader trust theme, see /blog/trust-gap-ai-assisted-coding. Install and catalog details: https://critiq.dev/products/oss.

What AI code review optimizes for

AI code review tools read diffs and natural language context, then produce summaries, suggestions, and sometimes security callouts in prose. They adapt to unfamiliar frameworks and can phrase feedback in a friendly voice. Tradeoffs: output varies with model and prompt versions, findings rarely cite a stable identifier, and reproduction often requires the vendor cloud rather than a CLI flag.

• Strengths: exploration, onboarding explanations, broad "does this look risky" skims
• Weaknesses at merge gates: non-determinism, weak severity semantics, hard audit trail
• Typical output: paragraphs without ruleId fields your policy can allowlist

What rule-based code review guarantees

Rule-based code review ties every finding to a catalog entry you can open: ts.security.hardcoded-credentials, py.security.sql-interpolation, and hundreds of others in @critiq/rules. The engine evaluates facts from source adapters; the rule file documents what matched, why it matters, and how to fix it. Same inputs and versions yield the same results in your terminal, JSON export, SARIF, and critiq-action inline comments.

npx @critiq/cli check .
critiq check --format json .

That reproducibility is what merge automation needs. Branch protection and fail-on-severity gates assume findings are comparable across runs, not reinterpreted by a model temperature setting.

Rule-based vs AI code review side by side

• Identity: rule id and catalog path vs paraphrased "potential issue"
• Reproduction: critiq check locally vs vendor-only re-run
• Severity: low through critical for policy vs subjective wording
• Evidence: matched fact and line vs summary without inspectable chain
• Change control: pin rules-version in CI vs model updates you do not ship

Neither replaces human review entirely. AI assistants in the editor can speed exploration; rule-based checks belong at the merge gate where accountability matters.

A concrete finding shape reviewers trust

Rule: ts.security.hardcoded-credentials
Severity: high
File: src/auth/signing.ts:14

Finding: Literal signing key in source.

Fix: Load the key from environment or your secrets manager.

That record is boring on purpose. Security leads can disable or downgrade ts.security.hardcoded-credentials in .critiq/config.yaml. New hires can read the rule text without asking whether a bot "meant it." Compare that to a thread that only says "consider rotating credentials" with no link to policy.

Wire rule-based review into GitHub

Post the same findings on pull requests with critiq-action so authors see rule IDs on the diff. Setup walkthrough: https://critiq.dev/integrations/github-actions.

      - uses: critiq-dev/critiq-action@v1
        with:
          comment-mode: inline
          fail-on-severity: high

Start with fail-on-severity: off if the team is learning the catalog; raise the gate when high-severity security rules are trusted. Dedupe keeps reruns from spamming resolved threads.

Using both without mixing trust models

A practical split: use AI assistants while authoring for drafts, tests, and explanations. Use rule-based review for merge policy on security, correctness, and catalog-backed quality signals. Do not treat fluent PR prose as a failing check unless it maps to a named rule you can disable and audit.

Critiq hosted AI review is on the longer roadmap; the shipped foundation is inspectable rules and local-first CLI checks. Trust at merge time starts there.

Next steps

• Trust gap in AI-assisted coding, /blog/trust-gap-ai-assisted-coding
• Evidence over vibes, /blog/evidence-over-vibes-code-review
• OSS CLI and rules, https://critiq.dev/products/oss
• GitHub Actions, https://critiq.dev/integrations/github-actions

Choose rule-based review when findings must be named, versioned, and reproducible. Treat AI summaries as optional context, not infrastructure. That division keeps merge gates honest while assistants stay useful in the editor.