Module: S06 — Secure Code Review Harnesses · Duration: ~75 minutes
Format: Verbatim transcript. Cues [SLIDE N] map to 03-slide-deck.html.
[SLIDE 1 — Title]
This is Module S Zero Six: Secure Code Review Harnesses. Seventy-five minutes. The first module of Pillar Two — Application Security and the SDLC. Where the harness turns left, from offensive operations to defending the development pipeline. Four sub-sections: the code review harness architecture, the false positive problem, semantic codebase memory, and autofix with an approval gate. By the end you can build a harness that takes a pull request, runs a layered pipeline, triages findings with an LLM, retrieves cross-file context, and proposes fixes a human approves before merge.
[SLIDE 2 — S06.1: Code Review Harness Architecture]
Sub-section one. The code review harness architecture. The input is a change — a pull request diff, a full file, a repo snapshot, or a commit hook payload. The output is a structured set of findings: file, line, CWE, severity, confidence, remediation. The architecture between input and output is a layered pipeline. Each layer has a job. No single layer is trusted alone.
[SLIDE 3 — The layered pipeline]
The pipeline runs in a specific order. AST parsing first — syntax tree, symbol resolution, cheap structural checks. Semgrep second — fast pattern-based rules, custom and registry, high recall. CodeQL third — deep data-flow queries, taint tracking, higher precision but slower. The LLM semantic layer fourth — context-aware judgement, false positive triage. Synthesis fifth — dedup across scanners, normalize severity, assign confidence, route.
The order is load-bearing. Deterministic scanners run first because they are fast, deterministic, and free of hallucination. The LLM runs last, on the filtered set of raw findings, to do what deterministic scanners cannot.
[SLIDE 4 — Why deterministic first, LLM last]
Here is the anti-pattern teams fall into. They run the LLM over the raw diff to find vulnerabilities directly, skipping the deterministic layers. It is slow. It is expensive. It is inconsistent across runs. And it hallucinates — both false positives and missed real findings. The LLM's strength is judgement on a small bounded set of candidates, not brute-force scanning.
By the time the LLM sees a finding, the deterministic layers have already established that a pattern matches a known weakness. The LLM's job is to answer one question: is this actually exploitable here, given the surrounding code? That separation is measurable — each layer has a recall budget and a precision budget.
[SLIDE 5 — Routing]
Routing. The synthesis layer maps each finding to an action based on severity and confidence. Critical or high severity with high confidence — block the PR, the status check fails. Critical or high with medium confidence — queue for a human reviewer, post an inline comment. Medium or low with high confidence — auto-comment inline. Anything with low confidence — file silently in the dashboard, no PR noise.
The principle: blocking a PR is a strong action. It should require both high severity and high post-triage confidence. Over-blocking trains developers to click override without reading. The goal is not to gate every PR. It is to gate the findings you are confident are real and serious.
[SLIDE 6 — S06.2: The False Positive Problem]
Sub-section two. The false positive problem. Semgrep and CodeQL are recall machines. On a large codebase, Semgrep's registry rules produce hundreds of findings. A large fraction are false positives — the pattern matched, but the framework sanitizes the input, or the sink is not reachable from untrusted data, or the flagged function is test-only. Raw scanner output at scale is low precision. Surface it directly and developers ignore the bot within a week.
[SLIDE 7 — LLM-powered triage]
The fix is LLM-powered triage. For each raw finding, the harness assembles a triage prompt: the finding, the matched code, the surrounding function, and any retrieved cross-file context. The LLM is asked a bounded set of questions. Is the tainted input reachable from untrusted data in this code path? Is there a sanitizer between source and sink? Is this a true positive or a false positive? What confidence?
The model does not find the vulnerability — the deterministic scanner already did that. The model judges it, in context. This is the same reader-actor separation from S Zero One point three and the model-judged triage pattern from S Zero Two point four. The system that finds a finding should not be the system that scores it.
[SLIDE 8 — The feedback loop]
Triage is not set-and-forget. When a human reviewer marks a finding as a false positive, or confirms a true positive the model called low-confidence, that judgement becomes a labeled example. Those examples feed back as few-shot context for future triage of similar findings. The feedback store is a vector index over past human verdicts.
Over weeks, the triage prompt for "is this Django ORM call actually injectable" includes three prior human judgements that said no. And the model's precision on that class of finding improves. This is the loop that turns a static ruleset into a learning harness.
[SLIDE 9 — Measuring the harness]
The harness is measurable. Against a labeled vulnerability dataset — OWASP Benchmark for Java, Juliet for C/C++/Java, DVWA for PHP, or your own historically-labeled findings — you measure precision, recall, and F1.
Raw Semgrep might give you ninety-five percent recall but forty percent precision. After LLM triage at the high-confidence threshold, precision rises to around ninety-two percent while recall drops to around eighty-five. That trade is almost always worth it. A ninety-two over eighty-five pipeline is usable. A forty over ninety-five pipeline is noise that gets muted. Run the measurement on every harness change.
[SLIDE 10 — S06.3: Semantic Codebase Memory]
Sub-section three. Semantic codebase memory. Single-file analysis misses cross-file vulnerabilities. A SQL query is built in repository dot py. The tainted input enters through api handler dot py. The sink executes in db engine dot py. A Semgrep rule on repository dot py sees the query but not the source. A rule on handler dot py sees the source but not the sink. The taint flow crosses file boundaries. No single-file scan can see it whole.
[SLIDE 11 — Function-level indexing]
The solution is function-level semantic indexing. Each function is embedded and stored in a vector database with metadata — file, symbol, callers, callees, and the raw source. For a finding at file X line Y, the retrieval query asks the index for the enclosing function, all callers — where the data comes from — all callees — where the data goes, the sink — and the definitions of sanitizer-like functions.
Retrieval is a mix of graph traversal through the static call graph and semantic search via vector similarity. The index is rebuilt incrementally on each PR — only changed functions and their callers and callees are re-embedded.
[SLIDE 12 — Context budget]
The retrieved context cannot all fit in the triage prompt. A finding deep in a call graph can pull in dozens of functions, each hundreds of lines. The harness enforces a context budget — a token limit, typically four to eight thousand tokens. Within the budget, retrieval is prioritized. Direct callers and callees first — they define the taint path. Sanitizer-like functions next — they resolve the false-positive-or-true-positive question. Transitive callers last — background context.
The budget is a tunable. Too small and you miss the sanitizer two frames up the stack. Too large and the prompt gets expensive and the model's attention dilutes. Calibrate against the labeled dataset.
[SLIDE 13 — S06.4: Autofix with Approval Gate]
Sub-section four. Autofix with an approval gate. The final layer of a mature review harness generates a fix for each confirmed finding — not just a comment describing the problem. But a fix that a model proposes and a fix that ships to production are separated by a verification and approval gate that is non-negotiable.
The loop runs in order. A confirmed finding triggers patch generation — a minimal, targeted diff. The patch passes four automated gates. Linter clean. Semgrep re-scan clean — the specific rule no longer matches, and no new findings are introduced. Test suite passes with coverage preserved. Patch quality score above threshold. Only patches passing all four gates open a draft PR.
[SLIDE 14 — Why approval is non-negotiable]
Then the non-negotiable gate. Human approval. LLM-generated patches can introduce new vulnerabilities while fixing the reported one. A model that closes a SQL injection by escaping quotes may miss a second injection in the same function. A model that adds an authorization check may place it after the sensitive operation. The deterministic gates catch some of these, but not all. The approval gate exists for the rest.
The PR is the approval mechanism. The harness opens a draft PR — not a direct commit, not an auto-merge. The developer reviews the patch against the finding and the verification evidence, requests changes or approves, and merges. There is constant pressure to quote "just auto-merge the high-confidence ones" — it saves a click, it feels like progress. Resist it. The cost of one auto-merged regression that takes down production or introduces a CVE outweighs the saved clicks across thousands of safe merges. The approval gate is held by a human.
[SLIDE 15 — What you take into S07]
S Zero Seven shifts from code to architecture. The code review harness is operational — layered pipeline, false-positive triage, semantic memory, approval-gated autofix. Threat modeling harnesses take the same principles upstream: ingest draw dot io, Terraform, OpenAPI into a data flow diagram, run STRIDE analysis, generate mitigations and issue tickets. From reviewing code to reviewing architecture.
[End of script]
# Teaching Script — Module S06: Secure Code Review Harnesses **Module**: S06 — Secure Code Review Harnesses · **Duration**: ~75 minutes **Format**: Verbatim transcript. Cues `[SLIDE N]` map to `03-slide-deck.html`. --- [SLIDE 1 — Title] This is Module S Zero Six: Secure Code Review Harnesses. Seventy-five minutes. The first module of Pillar Two — Application Security and the SDLC. Where the harness turns left, from offensive operations to defending the development pipeline. Four sub-sections: the code review harness architecture, the false positive problem, semantic codebase memory, and autofix with an approval gate. By the end you can build a harness that takes a pull request, runs a layered pipeline, triages findings with an LLM, retrieves cross-file context, and proposes fixes a human approves before merge. [SLIDE 2 — S06.1: Code Review Harness Architecture] Sub-section one. The code review harness architecture. The input is a change — a pull request diff, a full file, a repo snapshot, or a commit hook payload. The output is a structured set of findings: file, line, CWE, severity, confidence, remediation. The architecture between input and output is a layered pipeline. Each layer has a job. No single layer is trusted alone. [SLIDE 3 — The layered pipeline] The pipeline runs in a specific order. AST parsing first — syntax tree, symbol resolution, cheap structural checks. Semgrep second — fast pattern-based rules, custom and registry, high recall. CodeQL third — deep data-flow queries, taint tracking, higher precision but slower. The LLM semantic layer fourth — context-aware judgement, false positive triage. Synthesis fifth — dedup across scanners, normalize severity, assign confidence, route. The order is load-bearing. Deterministic scanners run first because they are fast, deterministic, and free of hallucination. The LLM runs last, on the filtered set of raw findings, to do what deterministic scanners cannot. [SLIDE 4 — Why deterministic first, LLM last] Here is the anti-pattern teams fall into. They run the LLM over the raw diff to find vulnerabilities directly, skipping the deterministic layers. It is slow. It is expensive. It is inconsistent across runs. And it hallucinates — both false positives and missed real findings. The LLM's strength is judgement on a small bounded set of candidates, not brute-force scanning. By the time the LLM sees a finding, the deterministic layers have already established that a pattern matches a known weakness. The LLM's job is to answer one question: is this actually exploitable here, given the surrounding code? That separation is measurable — each layer has a recall budget and a precision budget. [SLIDE 5 — Routing] Routing. The synthesis layer maps each finding to an action based on severity and confidence. Critical or high severity with high confidence — block the PR, the status check fails. Critical or high with medium confidence — queue for a human reviewer, post an inline comment. Medium or low with high confidence — auto-comment inline. Anything with low confidence — file silently in the dashboard, no PR noise. The principle: blocking a PR is a strong action. It should require both high severity and high post-triage confidence. Over-blocking trains developers to click override without reading. The goal is not to gate every PR. It is to gate the findings you are confident are real and serious. [SLIDE 6 — S06.2: The False Positive Problem] Sub-section two. The false positive problem. Semgrep and CodeQL are recall machines. On a large codebase, Semgrep's registry rules produce hundreds of findings. A large fraction are false positives — the pattern matched, but the framework sanitizes the input, or the sink is not reachable from untrusted data, or the flagged function is test-only. Raw scanner output at scale is low precision. Surface it directly and developers ignore the bot within a week. [SLIDE 7 — LLM-powered triage] The fix is LLM-powered triage. For each raw finding, the harness assembles a triage prompt: the finding, the matched code, the surrounding function, and any retrieved cross-file context. The LLM is asked a bounded set of questions. Is the tainted input reachable from untrusted data in this code path? Is there a sanitizer between source and sink? Is this a true positive or a false positive? What confidence? The model does not find the vulnerability — the deterministic scanner already did that. The model judges it, in context. This is the same reader-actor separation from S Zero One point three and the model-judged triage pattern from S Zero Two point four. The system that finds a finding should not be the system that scores it. [SLIDE 8 — The feedback loop] Triage is not set-and-forget. When a human reviewer marks a finding as a false positive, or confirms a true positive the model called low-confidence, that judgement becomes a labeled example. Those examples feed back as few-shot context for future triage of similar findings. The feedback store is a vector index over past human verdicts. Over weeks, the triage prompt for "is this Django ORM call actually injectable" includes three prior human judgements that said no. And the model's precision on that class of finding improves. This is the loop that turns a static ruleset into a learning harness. [SLIDE 9 — Measuring the harness] The harness is measurable. Against a labeled vulnerability dataset — OWASP Benchmark for Java, Juliet for C/C++/Java, DVWA for PHP, or your own historically-labeled findings — you measure precision, recall, and F1. Raw Semgrep might give you ninety-five percent recall but forty percent precision. After LLM triage at the high-confidence threshold, precision rises to around ninety-two percent while recall drops to around eighty-five. That trade is almost always worth it. A ninety-two over eighty-five pipeline is usable. A forty over ninety-five pipeline is noise that gets muted. Run the measurement on every harness change. [SLIDE 10 — S06.3: Semantic Codebase Memory] Sub-section three. Semantic codebase memory. Single-file analysis misses cross-file vulnerabilities. A SQL query is built in repository dot py. The tainted input enters through api handler dot py. The sink executes in db engine dot py. A Semgrep rule on repository dot py sees the query but not the source. A rule on handler dot py sees the source but not the sink. The taint flow crosses file boundaries. No single-file scan can see it whole. [SLIDE 11 — Function-level indexing] The solution is function-level semantic indexing. Each function is embedded and stored in a vector database with metadata — file, symbol, callers, callees, and the raw source. For a finding at file X line Y, the retrieval query asks the index for the enclosing function, all callers — where the data comes from — all callees — where the data goes, the sink — and the definitions of sanitizer-like functions. Retrieval is a mix of graph traversal through the static call graph and semantic search via vector similarity. The index is rebuilt incrementally on each PR — only changed functions and their callers and callees are re-embedded. [SLIDE 12 — Context budget] The retrieved context cannot all fit in the triage prompt. A finding deep in a call graph can pull in dozens of functions, each hundreds of lines. The harness enforces a context budget — a token limit, typically four to eight thousand tokens. Within the budget, retrieval is prioritized. Direct callers and callees first — they define the taint path. Sanitizer-like functions next — they resolve the false-positive-or-true-positive question. Transitive callers last — background context. The budget is a tunable. Too small and you miss the sanitizer two frames up the stack. Too large and the prompt gets expensive and the model's attention dilutes. Calibrate against the labeled dataset. [SLIDE 13 — S06.4: Autofix with Approval Gate] Sub-section four. Autofix with an approval gate. The final layer of a mature review harness generates a fix for each confirmed finding — not just a comment describing the problem. But a fix that a model proposes and a fix that ships to production are separated by a verification and approval gate that is non-negotiable. The loop runs in order. A confirmed finding triggers patch generation — a minimal, targeted diff. The patch passes four automated gates. Linter clean. Semgrep re-scan clean — the specific rule no longer matches, and no new findings are introduced. Test suite passes with coverage preserved. Patch quality score above threshold. Only patches passing all four gates open a draft PR. [SLIDE 14 — Why approval is non-negotiable] Then the non-negotiable gate. Human approval. LLM-generated patches can introduce new vulnerabilities while fixing the reported one. A model that closes a SQL injection by escaping quotes may miss a second injection in the same function. A model that adds an authorization check may place it after the sensitive operation. The deterministic gates catch some of these, but not all. The approval gate exists for the rest. The PR is the approval mechanism. The harness opens a draft PR — not a direct commit, not an auto-merge. The developer reviews the patch against the finding and the verification evidence, requests changes or approves, and merges. There is constant pressure to quote "just auto-merge the high-confidence ones" — it saves a click, it feels like progress. Resist it. The cost of one auto-merged regression that takes down production or introduces a CVE outweighs the saved clicks across thousands of safe merges. The approval gate is held by a human. [SLIDE 15 — What you take into S07] S Zero Seven shifts from code to architecture. The code review harness is operational — layered pipeline, false-positive triage, semantic memory, approval-gated autofix. Threat modeling harnesses take the same principles upstream: ingest draw dot io, Terraform, OpenAPI into a data flow diagram, run STRIDE analysis, generate mitigations and issue tickets. From reviewing code to reviewing architecture. --- [End of script]