Diagrams — Module S06: Secure Code Review Harnesses

{
  "name": "S06 — Code Review Harness Pipeline",
  "nodes": [
    { "name": "PR Diff / File / Snapshot", "type": "n8n-nodes-base.manualTrigger", "position": [200, 300], "notes": "Pull request diff, full file, repo snapshot, or commit hook payload" },
    { "name": "AST Parsing", "type": "n8n-nodes-base.executeCommand", "position": [420, 300], "notes": "Syntax tree, symbol resolution, cheap structural checks" },
    { "name": "Semgrep", "type": "n8n-nodes-base.executeCommand", "position": [640, 200], "notes": "Fast pattern-based rules (custom + registry). High recall, low precision at scale." },
    { "name": "CodeQL", "type": "n8n-nodes-base.executeCommand", "position": [640, 400], "notes": "Deep data-flow queries, taint tracking. Higher precision, slower." },
    { "name": "Raw Findings (normalized)", "type": "n8n-nodes-base.set", "position": [860, 300], "notes": "file, line, CWE, severity, source. Dedup key assigned." },
    { "name": "Semantic Codebase Memory", "type": "n8n-nodes-base.set", "position": [1080, 450], "notes": "Function-level vector index. For each finding: retrieve enclosing fn, callers, callees, sanitizers." },
    { "name": "LLM Triage", "type": "n8n-nodes-base.set", "position": [1080, 200], "notes": "Per finding: reachable? sanitized? verdict + confidence. Few-shot from feedback store." },
    { "name": "Synthesis", "type": "n8n-nodes-base.set", "position": [1300, 300], "notes": "Dedup across scanners, normalize severity, assign confidence, route." },
    { "name": "Block / Queue / Comment", "type": "n8n-nodes-base.set", "position": [1520, 300], "notes": "High+critical → block PR. Med → queue for human. Low → file silently." },
    { "name": "Feedback Store", "type": "n8n-nodes-base.set", "position": [1080, 100], "notes": "Human verdicts on past findings → few-shot examples for future triage" }
  ],
  "connections": {
    "PR Diff / File / Snapshot": { "main": [[{ "node": "AST Parsing", "type": "main", "index": 0 }]] },
    "AST Parsing": { "main": [[{ "node": "Semgrep", "type": "main", "index": 0 }], [{ "node": "CodeQL", "type": "main", "index": 0 }]] },
    "Semgrep": { "main": [[{ "node": "Raw Findings (normalized)", "type": "main", "index": 0 }]] },
    "CodeQL": { "main": [[{ "node": "Raw Findings (normalized)", "type": "main", "index": 0 }]] },
    "Raw Findings (normalized)": { "main": [[{ "node": "LLM Triage", "type": "main", "index": 0 }]] },
    "Semantic Codebase Memory": { "main": [[{ "node": "LLM Triage", "type": "main", "index": 0 }]] },
    "Feedback Store": { "main": [[{ "node": "LLM Triage", "type": "main", "index": 0 }]] },
    "LLM Triage": { "main": [[{ "node": "Synthesis", "type": "main", "index": 0 }]] },
    "Synthesis": { "main": [[{ "node": "Block / Queue / Comment", "type": "main", "index": 0 }]] }
  }
}

flowchart TD
    RAW["Raw scanner output<br/>Semgrep + CodeQL on large codebase<br/>~95% recall, ~40% precision"]
    NOTE1["Hundreds of findings<br/>majority are false positives<br/>(framework sanitizes, test-only code, unreachable sink)"]
    TRIAGE["LLM triage per finding<br/>+ cross-file context<br/>+ feedback few-shot"]
    RESULT["High-confidence findings only<br/>~85% recall, ~92% precision"]

    RAW --> NOTE1
    NOTE1 --> TRIAGE
    TRIAGE --> RESULT

    BAD["Surface raw findings directly"]
    BAD --> MUTE["Developer mutes the bot within a week"]

    style RAW fill:#2a1810,stroke:#a04000,color:#f0a868
    style TRIAGE fill:#14141f,stroke:#5eead4,color:#5eead4
    style RESULT fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style BAD fill:#2a0d0d,stroke:#a00000,color:#f08080
    style MUTE fill:#2a0d0d,stroke:#a00000,color:#f08080

flowchart LR
    FINDING["Raw finding<br/>file, line, CWE, snippet"]
    ENCLOSING["Enclosing function<br/>(from AST)"]
    SEMANTIC["Semantic memory<br/>callers, callees, sanitizers<br/>(within context budget)"]
    FEEDBACK["Feedback store<br/>past human verdicts<br/>as few-shot examples"]
    PROMPT["Triage prompt"]
    MODEL["LLM triage"]
    OUTPUT["verdict: tp/fp<br/>confidence: high/med/low<br/>reasoning"]

    FINDING --> PROMPT
    ENCLOSING --> PROMPT
    SEMANTIC --> PROMPT
    FEEDBACK --> PROMPT
    PROMPT --> MODEL --> OUTPUT

    style FINDING fill:#0d1b2a,stroke:#1b4f72,color:#e4e4e8
    style SEMANTIC fill:#14141f,stroke:#5eead4,color:#5eead4
    style FEEDBACK fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style MODEL fill:#14141f,stroke:#5eead4,color:#5eead4

flowchart TD
    FIND["Finding: SQL query in<br/>repository.py:42"]
    ENC["Retrieve: enclosing fn<br/>get_user()"]
    CALLERS["Retrieve: callers of get_user()<br/>handle_login() in api/handler.py"]
    CALLEES["Retrieve: callees<br/>db.execute() — the sink"]
    SANI["Retrieve: sanitizer-like fns<br/>validate_input() in validators.py"]
    BUDGET{"Within context<br/>budget (4-8k tokens)?"}
    PROMPT["Assemble triage prompt<br/>source → sanitizer → sink"]
    VERDICT["Verdict: FP<br/>(validate_input escapes<br/>before reaching sink)"]

    FIND --> ENC
    ENC --> CALLERS
    ENC --> CALLEES
    ENC --> SANI
    CALLERS --> BUDGET
    CALLEES --> BUDGET
    SANI --> BUDGET
    BUDGET -->|"yes, prioritized"| PROMPT --> VERDICT
    BUDGET -->|"no, truncate lowest-priority"| PROMPT

    style FIND fill:#2a1810,stroke:#a04000,color:#f0a868
    style SANI fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style BUDGET fill:#14141f,stroke:#5eead4,color:#5eead4
    style VERDICT fill:#0d2818,stroke:#1e8449,color:#82e0aa

flowchart TD
    CONF["Confirmed finding<br/>(high confidence, post-triage)"]
    GEN["LLM generates patch<br/>minimal, targeted diff"]
    LINT["Gate 1: linter clean?"]
    SEM["Gate 2: Semgrep re-scan<br/>finding resolved? no NEW findings?"]
    TEST["Gate 3: test suite passes<br/>coverage preserved?"]
    QUALITY["Gate 4: patch quality score<br/>above threshold?"]
    DRAFT["Open DRAFT PR<br/>patch + finding + verification evidence"]
    APPROVE["GATE 5: Human approval<br/>(non-negotiable)"]
    MERGE["Merge"]
    HOLD["Hold back<br/>file as 'fix suggested,<br/>needs human authorship'"]

    CONF --> GEN --> LINT
    LINT -->|"fail"| DISCARD["Discard / revise patch"]
    LINT -->|"pass"| SEM
    SEM -->|"fail"| DISCARD
    SEM -->|"pass"| TEST
    TEST -->|"fail"| DISCARD
    TEST -->|"pass"| QUALITY
    QUALITY -->|"below threshold"| HOLD
    QUALITY -->|"above"| DRAFT --> APPROVE
    APPROVE -->|"approve"| MERGE
    APPROVE -->|"request changes"| GEN

    style CONF fill:#0d1b2a,stroke:#1b4f72,color:#e4e4e8
    style GEN fill:#14141f,stroke:#5eead4,color:#5eead4
    style APPROVE fill:#2a0d0d,stroke:#a00000,color:#f08080
    style MERGE fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style DISCARD fill:#2a0d0d,stroke:#a00000,color:#f08080
    style HOLD fill:#2a1810,stroke:#a04000,color:#f0a868

# Diagrams — Module S06: Secure Code Review Harnesses

> All Mermaid validated in Mermaid Live Editor. n8n JSON is structurally valid and importable.

---

## Diagram 1 — The Code Review Harness Pipeline (n8n)

**Type**: n8n workflow (importable JSON)
**Purpose**: The primary visual — the full layered pipeline from PR input to structured findings. Shows the deterministic layers (AST, Semgrep, CodeQL), the LLM triage layer, and synthesis, with the semantic codebase memory feeding context.

```json
{
  "name": "S06 — Code Review Harness Pipeline",
  "nodes": [
    { "name": "PR Diff / File / Snapshot", "type": "n8n-nodes-base.manualTrigger", "position": [200, 300], "notes": "Pull request diff, full file, repo snapshot, or commit hook payload" },
    { "name": "AST Parsing", "type": "n8n-nodes-base.executeCommand", "position": [420, 300], "notes": "Syntax tree, symbol resolution, cheap structural checks" },
    { "name": "Semgrep", "type": "n8n-nodes-base.executeCommand", "position": [640, 200], "notes": "Fast pattern-based rules (custom + registry). High recall, low precision at scale." },
    { "name": "CodeQL", "type": "n8n-nodes-base.executeCommand", "position": [640, 400], "notes": "Deep data-flow queries, taint tracking. Higher precision, slower." },
    { "name": "Raw Findings (normalized)", "type": "n8n-nodes-base.set", "position": [860, 300], "notes": "file, line, CWE, severity, source. Dedup key assigned." },
    { "name": "Semantic Codebase Memory", "type": "n8n-nodes-base.set", "position": [1080, 450], "notes": "Function-level vector index. For each finding: retrieve enclosing fn, callers, callees, sanitizers." },
    { "name": "LLM Triage", "type": "n8n-nodes-base.set", "position": [1080, 200], "notes": "Per finding: reachable? sanitized? verdict + confidence. Few-shot from feedback store." },
    { "name": "Synthesis", "type": "n8n-nodes-base.set", "position": [1300, 300], "notes": "Dedup across scanners, normalize severity, assign confidence, route." },
    { "name": "Block / Queue / Comment", "type": "n8n-nodes-base.set", "position": [1520, 300], "notes": "High+critical → block PR. Med → queue for human. Low → file silently." },
    { "name": "Feedback Store", "type": "n8n-nodes-base.set", "position": [1080, 100], "notes": "Human verdicts on past findings → few-shot examples for future triage" }
  ],
  "connections": {
    "PR Diff / File / Snapshot": { "main": [[{ "node": "AST Parsing", "type": "main", "index": 0 }]] },
    "AST Parsing": { "main": [[{ "node": "Semgrep", "type": "main", "index": 0 }], [{ "node": "CodeQL", "type": "main", "index": 0 }]] },
    "Semgrep": { "main": [[{ "node": "Raw Findings (normalized)", "type": "main", "index": 0 }]] },
    "CodeQL": { "main": [[{ "node": "Raw Findings (normalized)", "type": "main", "index": 0 }]] },
    "Raw Findings (normalized)": { "main": [[{ "node": "LLM Triage", "type": "main", "index": 0 }]] },
    "Semantic Codebase Memory": { "main": [[{ "node": "LLM Triage", "type": "main", "index": 0 }]] },
    "Feedback Store": { "main": [[{ "node": "LLM Triage", "type": "main", "index": 0 }]] },
    "LLM Triage": { "main": [[{ "node": "Synthesis", "type": "main", "index": 0 }]] },
    "Synthesis": { "main": [[{ "node": "Block / Queue / Comment", "type": "main", "index": 0 }]] }
  }
}
```

**Reading the diagram**: Left to right. A PR diff enters and is parsed to an AST. Two deterministic scanners — Semgrep (fast, pattern-based) and CodeQL (deep, data-flow) — run on the AST output. Their raw findings are normalized to one schema. Each finding then passes to the LLM triage layer, which is fed by two stores: the semantic codebase memory (cross-file context for the finding) and the feedback store (past human verdicts as few-shot examples). The triage layer assigns a verdict and confidence. Synthesis deduplicates across scanners, normalizes severity, and routes: block the PR, queue for human review, or file silently. Deterministic first, LLM last — the order is load-bearing.

---

## Diagram 2 — The False Positive Problem (precision/recall trade)

**Type**: Mermaid (flowchart)
**Purpose**: Shows why raw scanner output is unusable at scale and how LLM triage shifts the precision/recall curve.

```mermaid
flowchart TD
    RAW["Raw scanner output<br/>Semgrep + CodeQL on large codebase<br/>~95% recall, ~40% precision"]
    NOTE1["Hundreds of findings<br/>majority are false positives<br/>(framework sanitizes, test-only code, unreachable sink)"]
    TRIAGE["LLM triage per finding<br/>+ cross-file context<br/>+ feedback few-shot"]
    RESULT["High-confidence findings only<br/>~85% recall, ~92% precision"]

    RAW --> NOTE1
    NOTE1 --> TRIAGE
    TRIAGE --> RESULT

    BAD["Surface raw findings directly"]
    BAD --> MUTE["Developer mutes the bot within a week"]

    style RAW fill:#2a1810,stroke:#a04000,color:#f0a868
    style TRIAGE fill:#14141f,stroke:#5eead4,color:#5eead4
    style RESULT fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style BAD fill:#2a0d0d,stroke:#a00000,color:#f08080
    style MUTE fill:#2a0d0d,stroke:#a00000,color:#f08080
```

**Reading the diagram**: Raw scanner output has high recall (catches most weaknesses) but low precision (many false positives at scale). Surfacing it directly is the anti-pattern — the developer mutes the bot. The LLM triage layer, fed with cross-file context and few-shot examples from past human verdicts, trades a small amount of recall for a large gain in precision. The result: ~85% recall, ~92% precision — a pipeline developers trust rather than mute.

---

## Diagram 3 — The Triage Prompt Assembly (context sources)

**Type**: Mermaid (flowchart)
**Purpose**: Shows the three context sources that feed the LLM triage prompt for a single finding.

```mermaid
flowchart LR
    FINDING["Raw finding<br/>file, line, CWE, snippet"]
    ENCLOSING["Enclosing function<br/>(from AST)"]
    SEMANTIC["Semantic memory<br/>callers, callees, sanitizers<br/>(within context budget)"]
    FEEDBACK["Feedback store<br/>past human verdicts<br/>as few-shot examples"]
    PROMPT["Triage prompt"]
    MODEL["LLM triage"]
    OUTPUT["verdict: tp/fp<br/>confidence: high/med/low<br/>reasoning"]

    FINDING --> PROMPT
    ENCLOSING --> PROMPT
    SEMANTIC --> PROMPT
    FEEDBACK --> PROMPT
    PROMPT --> MODEL --> OUTPUT

    style FINDING fill:#0d1b2a,stroke:#1b4f72,color:#e4e4e8
    style SEMANTIC fill:#14141f,stroke:#5eead4,color:#5eead4
    style FEEDBACK fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style MODEL fill:#14141f,stroke:#5eead4,color:#5eead4
```

**Reading the diagram**: The triage prompt for a single finding assembles four inputs: the raw finding itself (from the deterministic scanner), the enclosing function (from the AST), cross-file context from the semantic codebase memory (callers, callees, sanitizers — within a token budget), and few-shot examples from the feedback store (past human verdicts on similar findings). The model judges reachability and returns a verdict with confidence. The model does not find the finding — the scanners did. The model judges it.

---

## Diagram 4 — Semantic Codebase Memory (cross-file taint retrieval)

**Type**: Mermaid (flowchart)
**Purpose**: Shows how a finding in one file pulls cross-file context through the call graph to resolve taint flows.

```mermaid
flowchart TD
    FIND["Finding: SQL query in<br/>repository.py:42"]
    ENC["Retrieve: enclosing fn<br/>get_user()"]
    CALLERS["Retrieve: callers of get_user()<br/>handle_login() in api/handler.py"]
    CALLEES["Retrieve: callees<br/>db.execute() — the sink"]
    SANI["Retrieve: sanitizer-like fns<br/>validate_input() in validators.py"]
    BUDGET{"Within context<br/>budget (4-8k tokens)?"}
    PROMPT["Assemble triage prompt<br/>source → sanitizer → sink"]
    VERDICT["Verdict: FP<br/>(validate_input escapes<br/>before reaching sink)"]

    FIND --> ENC
    ENC --> CALLERS
    ENC --> CALLEES
    ENC --> SANI
    CALLERS --> BUDGET
    CALLEES --> BUDGET
    SANI --> BUDGET
    BUDGET -->|"yes, prioritized"| PROMPT --> VERDICT
    BUDGET -->|"no, truncate lowest-priority"| PROMPT

    style FIND fill:#2a1810,stroke:#a04000,color:#f0a868
    style SANI fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style BUDGET fill:#14141f,stroke:#5eead4,color:#5eead4
    style VERDICT fill:#0d2818,stroke:#1e8449,color:#82e0aa
```

**Reading the diagram**: A finding in `repository.py` cannot be judged in isolation. The semantic memory retrieves the enclosing function, its callers (where the data comes from — `handle_login`), its callees (the sink — `db.execute`), and sanitizer-like functions in the call path (`validate_input`). Retrieval is bounded by a token budget; within the budget, direct callers/callees and sanitizers are prioritized over transitive context. With this cross-file view, the triage model can determine that `validate_input` escapes the data before it reaches the sink — a false positive that single-file analysis could never resolve.

---

## Diagram 5 — The Autofix Loop with Approval Gate

**Type**: Mermaid (flowchart)
**Purpose**: The end-to-end autofix loop. Shows every gate the patch passes before merge, and why the human approval gate is non-negotiable.

```mermaid
flowchart TD
    CONF["Confirmed finding<br/>(high confidence, post-triage)"]
    GEN["LLM generates patch<br/>minimal, targeted diff"]
    LINT["Gate 1: linter clean?"]
    SEM["Gate 2: Semgrep re-scan<br/>finding resolved? no NEW findings?"]
    TEST["Gate 3: test suite passes<br/>coverage preserved?"]
    QUALITY["Gate 4: patch quality score<br/>above threshold?"]
    DRAFT["Open DRAFT PR<br/>patch + finding + verification evidence"]
    APPROVE["GATE 5: Human approval<br/>(non-negotiable)"]
    MERGE["Merge"]
    HOLD["Hold back<br/>file as 'fix suggested,<br/>needs human authorship'"]

    CONF --> GEN --> LINT
    LINT -->|"fail"| DISCARD["Discard / revise patch"]
    LINT -->|"pass"| SEM
    SEM -->|"fail"| DISCARD
    SEM -->|"pass"| TEST
    TEST -->|"fail"| DISCARD
    TEST -->|"pass"| QUALITY
    QUALITY -->|"below threshold"| HOLD
    QUALITY -->|"above"| DRAFT --> APPROVE
    APPROVE -->|"approve"| MERGE
    APPROVE -->|"request changes"| GEN

    style CONF fill:#0d1b2a,stroke:#1b4f72,color:#e4e4e8
    style GEN fill:#14141f,stroke:#5eead4,color:#5eead4
    style APPROVE fill:#2a0d0d,stroke:#a00000,color:#f08080
    style MERGE fill:#0d2818,stroke:#1e8449,color:#82e0aa
    style DISCARD fill:#2a0d0d,stroke:#a00000,color:#f08080
    style HOLD fill:#2a1810,stroke:#a04000,color:#f0a868
```

**Reading the diagram**: A confirmed finding triggers patch generation. The patch passes four automated gates: linter clean, Semgrep re-scan clean (the finding resolved, no new findings introduced), test suite passes with coverage preserved, and a quality score above threshold. Only patches passing all four gates open a draft PR. Then the non-negotiable gate: human approval. The developer reviews the patch against the finding and verification evidence, requests changes, or merges. Auto-merging is an anti-pattern — the human is the last line of defense against patches that fix one vulnerability while introducing another. The loop is patch, verify, approve, merge — in that order, every time.