75 min · Layered Pipeline · False Positive Triage · Semantic Memory · Autofix Gate
Prerequisite: S00–S05
A layered pipeline. Each layer has a job. No single layer is trusted alone.
Deterministic first, LLM last. Scanners give recall; the LLM gives precision.
LLM-first scanning is slow, expensive, inconsistent, and hallucination-prone. The LLM's strength is judgement on a small bounded set — not brute-force scanning.
By the time the LLM sees a finding, the deterministic layers have established the pattern matches. The LLM answers: is this exploitable HERE?
| Severity | Confidence | Action |
|---|---|---|
| Critical/High | High | Block PR (status check fails) |
| Critical/High | Medium | Queue for human; inline comment |
| Medium/Low | High | Auto-comment inline |
| Any | Low | File silently; no PR noise |
Over-blocking trains developers to click override without reading.
Semgrep + CodeQL are recall machines. On a large codebase: ~95% recall, ~40% precision. Surface raw output and developers mute the bot within a week.
Per finding: pass code + surrounding function + cross-file context. Ask: reachable from untrusted data? sanitized? true or false positive?
The model does not FIND the vulnerability — the scanner did. The model JUDGES it. Same reader/actor separation as S01.3, S02.4.
This is the loop that turns a static ruleset into a learning harness.
~92% / ~85%
Precision / Recall target after LLM triage (vs ~40% / ~95% raw).
| Stage | Precision | Recall |
|---|---|---|
| Raw Semgrep | ~40% | ~95% |
| After LLM triage | ~92% | ~85% |
Dataset: OWASP Benchmark, Juliet, DVWA, or your own labeled history.
Single-file analysis misses cross-file vulns. Taint enters in handler.py, sink executes in repository.py. No single-file scan sees the flow.
Each function embedded with metadata: file, symbol, callers, callees, source. Vector DB with a static call graph overlay.
For a finding at X:Y, retrieve: enclosing function, all callers (source), all callees (sink), sanitizer-like functions.
Retrieval = graph traversal (call graph) + semantic search (vector similarity).
Retrieved context cannot all fit. Enforce a token budget (4k–8k). Prioritize: direct callers/callees > sanitizers > transitive context.
Too small: miss the sanitizer two frames up. Too large: expensive, attention dilutes. Calibrate against labeled data.
LLM patches can introduce new vulnerabilities while fixing the reported one. Close an injection, miss a second one in the same function. Add an authz check after the sensitive op.
The PR is the approval mechanism. Draft PR, human merge, always. Auto-merging is an anti-pattern that will eventually ship a regression or a new CVE.
The code review harness is operational: layered pipeline, FP triage, semantic memory, approval-gated autofix.
S07 shifts from code to architecture: threat modeling harnesses that ingest draw.io/Terraform/OpenAPI into DFDs and run STRIDE analysis.