Blue Team of One field notes · security

HomeDetection EngineeringSentinel, end to end

SIEM architecture · part 2 of a series

How Sentinel detects: rules, alerts, incidents, and the hunt

Part 1 left the warehouse full of well-sorted data — and it still hadn't caught anyone. Storage isn't detection. This is the part that turns tables into a SIEM: the rules that read the data, the alerts they raise, the incidents you assemble from them, and the two other stances — hunting and entity analysis — that catch what the rules were never written to see.

Connecting a source and watching its table fill up feels like progress, and it's the moment most people believe they're "covered." They aren't. A full table detects nothing on its own — it's raw material. Detection is a separate act: something has to read that data and decide "this is worth a human's attention." In Sentinel there are three different stances you can take toward the same tables, and a mature practice runs all three.

Mental model

Rules are the smoke detectors you wired to the ceiling — they answer questions you already knew to ask, because you installed them. Hunting is walking the building at 2 a.m. with a flashlight, looking for the thing no detector covers. And the entity page is what you do once an alarm goes off: stop asking "which sensor tripped" and start asking "what is this person actually doing." Reactive, proactive, investigative — same building, three ways of looking.

TABLES stored data ANALYTICS RULE reads data on a schedule, decides what matters ALERT a single firing INCIDENT the case file — grouped alerts → response (Part 3) HUNTING · proactive questions you didn't know to ask — a find graduates into a rule promote ENTITY PAGE / UEBA · investigative flip the axis: not "what fired" but "who's involved" pivot THREE STANCES TOWARD THE SAME DATA
The detection engine. The horizontal spine — table → rule → alert → incident — is what most people mean by "detection." Hunting feeds it from above (find something, promote it to a rule); the entity page cuts in from below (pivot from the alert to the whole story of a user or host).

01Rules: the templates trap, and why a rule is more than its query

Two buckets — Microsoft's templates and your own — and one misconception that leaves teams with a green connector and zero live detections.

When you add a connector, its detection content arrives from the Content Hub as part of a solution — but it arrives as rule templates, not running rules. You still have to create a rule from a template and switch it on. This is the single most common beginner misread: the connector is green, data is flowing, and nothing is actually watching it because no rule was ever activated. Data flowing is not detection running.

Custom rules are where the craft lives — you write the KQL, decide what a single result means, and shape how findings behave. But here's the part that separates a working rule from a noisy one: the query is maybe half the rule. Every rule is its KQL plus a schedule, a threshold, entity mapping, and grouping logic — and the last two are what beginners skip.

ONE ANALYTICS RULE KQL QUERY the logic — what "bad" looks like …but only ~half the rule SCHEDULE how often it runs · lookback THRESHOLD how many hits before it fires ENTITY MAPPING tie results to user / host / IP GROUPING how alerts fold into incidents skip ENTITY MAPPING → alerts you can't pivot on skip GROUPING → a hundred duplicate incidents
A rule is an assembly, not a query. Entity mapping is what makes an alert investigable and correlatable; grouping is what keeps one noisy signal from burying the queue. Write perfect KQL and skip these two, and you've built a detection nobody can actually use.
SigninLogs
| where ResultType == 0
| where RiskLevelDuringSignIn == "high"
| where AppDisplayName has "Azure Portal"
// the query is the easy part. the rule also needs:
//   schedule  → run every 5m, look back 5m
//   threshold → fire on >= 1 result
//   entities  → map Account=UserPrincipalName, IP=IPAddress  (so you can pivot)
//   grouping  → fold repeats for the same Account into one incident

02The four kinds of rule

"Analytics rule" is a small family, and one member is where the Defender-unification story quietly shows up.

TypeWhat it doesReach for it when
ScheduledYour KQL on a timer — you set frequency and lookback. The workhorse.Almost always — this is what "a detection" usually means
Near-real-time (NRT)Same idea, runs about once a minute with minimal delay; comes with constraints (a single query, limited lookback).The handful of signals you can't wait ~5 minutes on
Microsoft SecurityTurns alerts from other Microsoft security products into Sentinel incidents — it relays, it doesn't query tables.Surfacing Defender / Defender for Cloud alerts as incidents
Anomaly / MLBehavioural baselines and multistage correlation instead of logic you wrote.Catching deviations you can't express as a query

That third row is the seam we argued about in the foundations post: with Sentinel and Defender unified, a growing share of what used to be your custom detection now arrives pre-built as Defender alerts, flowing in through Microsoft Security rules. The honest read of "what's still distinctly SIEM" is the Scheduled and Anomaly rows — the custom logic for the sources Defender doesn't natively understand — plus the retention layer underneath.

03Watchlists: the context you bring to the data

Connectors bring in what happened. Watchlists hold what you know. Crossing the two is where detection gets sharp.

A watchlist is a small, admin-curated lookup you import and maintain by hand — VIPs, known-good IP ranges, service-account inventory, terminated-employee lists, asset criticality. In a rule it does two opposite jobs: enrich/prioritise ("new-country sign-in by a user on the VIP list" — now it's worth waking someone) and filter/allowlist ("alert on admin logins except from IPs on my jump-box list"). Same join, in a workbook, turns anonymous rows into "that's the CFO's laptop."

A watchlist is reference data, not a log store

It's built for small, slow-changing, human-curated lists — not tens of thousands of high-churn rows. The test: if the data arrives on its own and grows, it belongs in a table; if you decide what's in it and it changes when you say so, it's a watchlist. And it's maintenance debt: a terminated-users watchlist nobody updates is worse than none — it silently stops catching the person it exists for, while everyone keeps trusting it. Keep them small, curated, and current, or they lie to you.

04Alert → incident, and grouping done honestly

The alert is a single fact. The incident is the story you assemble from related facts — and the art is drawing "related" tightly enough to be true.

The chain is precise: a rule runs, each qualifying result is an alert, and alerts are grouped into an incident — the unit a human actually works. Grouping is the right default, because without it one noisy source spawns a hundred near-identical incidents and the queue becomes useless. But what you group by decides whether grouping helps or hides.

✓ DEDUPE — right 1 incident same rule, 5 firings ✗ OVER-GROUP — hides a threat 1 incident — analyst triages, closes a 2nd, different attack (red) hidden inside ★ ASSEMBLE THE STORY — best AiTM sign-in rule inbox-rule created mass-mail sent 1 incident 3 rules · one mailbox = one attack Different rules, same actor, one window → assembled into a timeline, not scattered. Group by entity for a per-victim story; by rule to collapse a burst. Group aggressively enough to kill duplicates — never so broadly that a distinct attack hides in a case someone already looked at.
"Similar" is doing enormous work in "group similar alerts." Under-group and you drown in duplicates; over-group and a second, different attack rides inside an incident the analyst has already mentally closed. The best incidents pull different rules on one actor into a single timeline.

05Hunting: the questions you didn't know to ask

Rules catch attacks you predicted. Hunting is for the ones you didn't — and it's the R&D that feeds your detections.

An analytics rule can only fire on a pattern you already described. Hunting inverts that: you form a hypothesis ("if an attacker were living off the land here, what would it look like?"), run the query once against history, and chase the hunch. The part that closes the loop — and the reason hunting isn't just ad-hoc querying — is that a hunt which finds something real graduates into a scheduled rule. Hunting is where new detections are born.

Hypothesis"what would X look like?" Hunt over historyrun once, look back far Found something?real, repeatable pattern SCHEDULED RULEnow runs continuously
Hunt → confirm → promote. The loop is what makes hunting compound: every good hunt permanently widens your automated coverage instead of being a one-off search you forget.

When a hunt outgrows KQL — you need Python, a library, an ML model, or a graph you can't draw in a workbook — that's what notebooks are for: Jupyter notebooks you drive cell by cell. Two honest caveats: a notebook is a power tool you steer, not an agent that investigates for you; and it needs its own compute — you attach it to a separate Azure Machine Learning workspace (distinct from your Log Analytics workspace), which carries setup and cost. It's the specialist's instrument, used deliberately and rarely — not the daily driver.

06Entity behaviour: from "what fired" to "who's involved"

Mid-incident, the useful pivot isn't another query — it's a single screen that assembles everything tied to one actor.

The entity page takes a user, host, or IP and pulls together everything about it — incidents, alerts, sign-ins, an activity timeline — on one screen. It flips the axis of investigation from "what did this rule catch" to "what is this user doing," which is exactly the move you want once an alert points at someone. Underneath it, optionally, is UEBA (User and Entity Behaviour Analytics) — the ML baselining that flags "this account is acting anomalously versus its own history" and writes its own signal for the entity pages to surface. The entity page is always useful; UEBA is the optional engine that enriches it, and it carries cost — so it's a switch you flip deliberately, not a default.

The one-line model

Rules are known-question detection, hunting is unknown-question discovery, and entity pages are the pivot from "what fired" to "who's involved." Three stances toward the same data — and the rule is never just its KQL.

07Next: making it act

Detection ends with an incident sitting in a queue — accurately built, correctly grouped, ready for a human. But a well-made incident still hasn't done anything. Part 3 is response: automation rules and playbooks, the reflex and the muscle — including the single highest-leverage automation in a SOC, and the one place you should absolutely keep a human on the trigger.

Sources & further reading

Comments

Questions or corrections welcome. Sign in with GitHub to join the thread.