The nightly SOC pipeline

The full mechanics of the SOC pass: how a night of telemetry becomes a short, evidence-backed list a human can clear over coffee - entirely on the box, with a local 9B doing the reasoning and a deterministic shell keeping it honest.

The pass, stage by stage

One systemd timer at 04:00 runs the whole thing. Three deterministic shell stages wrap two LLM stages; the order matters - the guardrail runs first, and only what it defers ever reaches the model.

generate-security-report.sh · 04:00 nightly

[1-6] fetch Wazuh / Suricata / Fail2ban / VPN / CVE -> write report skeleton [7/8] soc-pass: llama-server up soc-l1 deterministic guardrail + no-think classify -> ## L1 Analysis soc-l2 thinking-high investigator (tools) -> ## L2 Review commit memory ; llama-server down [8/8] l2-review Claude-free Executive Summary (deterministic + one model call) 09:00 soc-notify parse Exec Summary -> Telegram digest

# the model server is started for the pass and torn down after; it is not a resident service

1. The deterministic floor

Before the model sees anything, a guardrail resolves what it can by rule: a sensitive action (new admin, cleared log, service install, promiscuous mode, new listener) is KNOWN only if a matching operator sec-note authorized it in the window - checked by code. No note, no authorization: it flags. The model cannot upgrade an unauthorized sensitive action to benign, because on the authorization question the model is never asked. Known-noise rules resolve the same way. Only the residue is deferred upward.

2. L1 - the classifier

A fast per-alert pass over only what the floor deferred. Temperature 0, output constrained by a GBNF grammar so the decoder can emit nothing but VERDICT: KNOWN|SUSPICIOUS|UNKNOWN. It is fed per-rule context pulled from the knowledge-base vault and a temporal signal (has this rule fired on recent days?). It closes the obvious and hands the rest to L2.

3. L2 - the investigator

The thinking tier. It reviews the whole report, item by item - not just the flagged findings - with a fixed set of read-only tools. For each item it reads the raw event, looks it up, states what it actually is, and classifies it. It does not punt: a benign item is confirmed with the evidence and dropped out of the escalation path (its severity lowered); anything escalated must carry the most-likely-benign reading, the most-likely-malicious reading, and the single next diagnostic step. Verbose where something is wrong; one line where it is not.

The tools

L2 has no shell. Its tools are fixed, parameterised functions - a model-supplied string is validated against a strict pattern before it ever reaches a command, and is never interpolated into a shell line.

Tool	Access	What it does
`secnote_query`	read	the operator authorization log - was this activity a pre-authorized operator action?
`dpkg_query`	read	Project0 package install/upgrade history - did a package update actually run?
`vault_lookup`	read	the SOC knowledge base - known-good patterns, alert-pattern notes, machine profiles
`wazuh_query`	read	raw events by rule / agent / window: the `full_log`, Windows process image / parent / hash, and any CTI verdict already attached to the alert
`memory_note`	write	propose a durable known-good pattern - structured, evidence-gated, quarantined (see below)
`cti_lookup`	read (external)	on-demand reputation for an unknown IOC: hash → VirusTotal, IP → AbuseIPDB, any → ThreatFox

Whole-report, per item

A good analyst does not write a paragraph justifying every benign known-noise rule - they note it in a line and spend their words where something is wrong. So does this:

## L2 Review · verbose on the anomaly, one line on the rest

FINDING: Rule 100160 on wazuh-server DECISION: CONFIRM-BENIGN EVIDENCE: wazuh_query log=system,critical,info cloud change time ... => RouterOS cloud NTP sync logged under the "critical" topic; matches the vault known-good pattern. All other top rules are known-noise: Suricata DNS visibility, MikroTik firewall drops, SELinux AVC checks, AF-PACKET capture artefacts. Nothing escalated. POSTURE: NORMAL

# every item is accounted for; only the one worth looking at gets the full block

Threat intel, two ways

The lab already runs a three-layer CTI enrichment (abuse.ch feeds, VirusTotal, AbuseIPDB). L2 taps it two ways: passively, because that enrichment writes its verdict back onto the alert, and wazuh_query surfaces it; and on demand, when it meets a genuinely unknown hash, IP, or domain and wants a reputation. Only a value that passes a strict hash / IP / domain check ever leaves the box, and only to the three intel APIs.

cti_lookup · identifying the unknown (EICAR test file)

cti_lookup --ioc 3395856ce81f2b7382dee72602f798b642f14140

VirusTotal: 65 malicious / 0 suspicious (EICAR-Test-File, W32.EicarTest.Trojan) ThreatFox: no_result

# a hash the analyst has never seen, answered against real threat intel - strong ESCALATE evidence

4. The Executive Summary - Claude-free

This step used to be a headless call to a cloud model. It is now local. The Overall Posture, Finding Verdicts, and Action Items are computed deterministically from L2's own output (posture follows the escalations; action items include actionable CVE patches). A single bounded local-model call writes only the prose Threat Highlights, from a full-report digest - and if it runs long, a deterministic top-rules fallback fires instead. The model never sets a posture or a verdict; it phrases. The same rule as everywhere else in the pipeline, one layer up.

The security engineering

An LLM that reads security alerts is reading attacker-influenceable text - a log line can contain whatever an attacker managed to get logged. That reframes the whole build around one question: how do you let a model help without letting the input drive it?

Untrusted input is fenced and defanged. Every alert body is wrapped in an explicit <untrusted_alert_data> fence and sanitized first - the characters an attacker would use to escape the fence (< >) or inject directives (` $) are stripped, so the payload stays trapped as inert data.
Benign needs positive evidence. Confirm-benign is allowed only on a concrete artifact the tools actually returned - a process image on a system path, a CTI verdict, a matching sec-note - never "it probably ran while I was working."
The one write tool is boxed. memory_note is structured (not free prose), evidence-gated, and quarantined to a pending section a human promotes. An attacker who gets a line logged cannot poison the durable known-good list into a permanent false-negative.
No shell, ever. Every tool is a fixed parameterised function; an IOC or package name is validated against a strict pattern before it touches a command.
Egress is a capability, not a default. The box is egress-hardened; the only outbound the analyst makes is a cti_lookup, and only the validated IOC leaves.
Fail-safe. If the L2 pass does not complete cleanly, posture defaults to CRITICAL - it fails loud, not silent.

What it is, and what it is not

It is a triage-and-investigation layer that keeps the security data in the lab, uses hardware that would otherwise idle overnight, and is built so untrusted input cannot steer it. It is not an autonomous responder and not the decision-maker: it narrows a day's noise down to what a human should actually look at, with cited evidence, and a person keeps every verdict. The whole system is Claude-free at run time - the code was built and is maintained with Claude Code over the vault MCP, but at 04:00 nothing leaves the box except a hash the analyst chose to check.