Security Stack - Deep Dive

The mechanics behind the security-stack overview: how telemetry is logged and ranked, how the high-severity few get pushed in real time, and the detection-engineering discipline that keeps the alert channel trustworthy. The overnight local-LLM triage that reads the long tail has its own writeup - the SOC pipeline.

How telemetry is logged

Everything converges on Wazuh as ranked alerts. Three sources feed it, and two things decide what actually matters.

Sources

• Endpoint agents on the core host and the client machines (workstation, laptop, the Windows lab VM) report auth, process, file-integrity, and Sysmon events.
• Edge syslog. The router (RouterOS) ships its logs to Wazuh with custom decoders and rules, so perimeter events (scans, drops) land next to host events.
• Suricata on the core writes network IDS alerts (eve.json) that Wazuh ingests - host, network, and perimeter detections share one timeline.

What rises to the top

• Rule level. Wazuh scores every alert; level 10+ is treated as high/critical, and that is what the triage pipeline reads.
• Threat-intel enrichment. Public IOC feeds (abuse.ch) are pulled into Wazuh CDB lists on a timer, so an alert touching a known-bad IP/domain/hash is tagged.

endpoint agents  ->
edge syslog      ->   Wazuh SIEM   ->   ranked alerts (level 10+ = high/crit)
suricata (eve)   ->        ^
abuse.ch feeds --(CDB)-----+

Real-time alerts

Not everything waits for the morning digest. Wazuh pushes level-7-and-up alerts straight to a Telegram bot (Hermes) the moment they fire - so a scan, a brute-force attempt, or a positive threat-intel hit reaches my phone in seconds, with the agent, the rule, and a one-line description.

Severity is the filter, and it is tuned deliberately:

Keeping the bar at level 7 is what makes the channel trustworthy: custom rules demote known-benign noise below 7 (still searchable, just quiet) and promote what matters to 7+. Some alerts are also enriched before they fire - a raw network detection is correlated with DNS and device data, so the message names the actual host and domain rather than just an IP.

The custom-rule work was not premature polish - alert fatigue was real. Out of the box the channel was a wall of false positives, and a channel that cries wolf is one you stop reading - which defeats the entire point of a real-time alert. So the rule-tuning is not decoration; it is what decides whether a level-7 ping on my phone actually means look now. The suppression discipline below came straight out of that same fight to make escalations trustworthy.

A narrow class of message bypasses the severity filter entirely: operational-availability alerts. When the LAN's DNS fails over - the core's resolver goes down and the edge router swaps to public resolvers - and again when it is restored, Hermes pushes a notification regardless of level, because "did my DNS just fail over?" is something I want to know the instant it happens, not have demoted as low-severity.

The failover is also logged with a marker - and that marker powers the real detection win. A custom rule watches for any change to the router's DNS configuration and, when the failover marker is absent, fires a high-severity alert. The expected operational change identifies itself by its marker; the same change without one - something quietly repointing the whole network's DNS - is flagged as a likely DNS hijack. One pair of rules turns a noisy operational event into a precise tripwire: the benign case is whitelisted by the marker it emits, so anything that looks like it but does not announce itself stands out. Failover mechanics in Network.

So there are two alerting tiers: Telegram in real time for the high-severity few, and the overnight LLM digest (below) for the long tail.

How the daily report is built

A scheduled overnight job runs a report generator that turns the last day of SIEM data into a single dated Markdown report (one per day, kept in the knowledge-base vault). The report always has the same skeleton, so the rest of the pipeline can parse it:

• Top Wazuh Rules - what fired most, with counts;
• High / Critical Alerts - the level-10+ events, per agent;
• Top Suricata Signatures - the loudest network detections.

That skeleton is the substrate for the overnight pass. A monthly roll-up runs as well.

Overnight triage, handed off

The report is the input to the overnight run: a local model works the long tail - the ~160k events a day this box sees - behind a deterministic guardrail that makes every authorization call in code, and escalates only what it cannot resolve to a human. It runs entirely on the box, with no cloud model in the loop. That pipeline - the guardrail floor, the L1 classifier, the L2 investigator and its read-only tools, and how the whole thing is hardened against the attacker-controlled text it has to read - is its own writeup: SOC Operations, with the full pipeline mechanics a click deeper.

Detection engineering: a suppression done right

The hard part of running detection is not catching things - it is not drowning in what you catch, and the difference is in how carefully you cut the noise. One machine makes the point. A Windows endpoint in the lab runs an anti-malware service (Malwarebytes) that fires constantly and legitimately - a steady stream of process events that are all benign and all noise. Left alone, that one host buries the handful of alerts that actually matter under its own routine chatter.

The reflex is to write a suppression rule and move on. The whole game is in how you scope it. A rule that matches the process by filename alone suppresses any process with that name, wherever it runs from - so an attacker who names a payload MBAMService.exe (or TrustedInstaller.exe, TiWorker.exe) and drops it in %TEMP% or AppData inherits the suppression for free. A fix for noise quietly becomes a living-off-the-land blind spot - a named gap in your own detection.

The discipline is to anchor every suppression to the binary's full install path, so only the real thing matches and a lookalike anywhere else still trips the alert:

<!-- WRONG - name only, any path matches -->
<field name="win.eventdata.image" type="pcre2">(?i)(MBAMService|mbam)\.exe</field>

<!-- RIGHT - full path anchor, only the real binary matches -->
<field name="win.eventdata.image" type="pcre2">(?i)^C:\\Program Files\\Malwarebytes\\Anti-Malware\\(MBAMService|mbam)\.exe$</field>

The principle. A suppression is a hole you punch in your own detection - cut it to the exact shape of the legitimate thing, precision over breadth, or you have handed an attacker a gap to walk through. Noise reduction and coverage pull against each other, and the way out is always to be more specific, never more broad. Two smaller traps live in the same rule: PCRE2 in Wazuh wants doubled backslashes in the path, and the ^...$ anchors are what stop a substring from matching mid-path.

Recreate it

The shape, not the secrets - templates land in the public repo (link to follow):

1. SIEM core. Stand up Wazuh (manager + indexer + dashboard). On modest hardware this is the deliberate pick over ELK - it carries the SIEM, agent management, and rule engine in a footprint a shared box can actually spare.
2. Network IDS. Run Suricata on the host, write eve.json, and have Wazuh ingest it so host and network detections share one timeline. Raise the AF-PACKET ring-size past the default if you see kernel drops under load.
3. Telemetry in. Install agents on every endpoint; ship the edge router's syslog with custom decoders/rules so perimeter events land next to host events.
4. Cut the noise deliberately. Demote known-benign rules below your alert threshold and promote what matters above it - and anchor every suppression to a full path, never a bare filename (see the suppression rule above). Enrich raw network alerts with DNS/device data so a message names a host, not an IP.
5. Threat-intel. Pull public IOC feeds into Wazuh CDB lists on a timer to tag alerts touching known-bad indicators.
6. Real-time + triage. Push level-7+ alerts to a chat bot for the few that need eyes now; run a local model overnight on the long tail, escalating only what it cannot resolve to a human.