Follow Detection validation is the practice of proving a detection rule still fires on the events it was written to catch. Detection decay is the silent failure of a rule that once worked, after a log source, schema, or parser changes underneath it. A rule that is deployed and enabled is not a rule that is proven to work.
How do you know a detection rule still works?
A rule works if it fires on a known event within a defined window. Everything else is an assumption.
Most teams treat “deployed and enabled” as the working state: the rule exists in the SIEM, it passes validation at deploy time, the coverage heatmap stays green. Execution status and coverage status both measure presence, not function. A rule that executes successfully against an empty result set looks identical, from the platform’s perspective, to a rule watching a quiet network. The dashboard stays green either way.
The honest evidence ladder has five tiers, weakest to strongest:
| Tier | State | What it proves | What it does NOT prove |
|---|---|---|---|
| 0 | Asserted | The rule exists | Nothing about function |
| 1 | Lint-passed | Parses correctly, required blocks present, fields exist in the target schema | That it matches anything |
| 2 | Fixture-replayed | Matches recorded known-bad events, rejects paired benign fixtures (offline) | That it fires in the live pipeline |
| 3 | Emulation-validated | Fires end to end against an emulated real procedure (Atomic Red Team or MITRE Caldera) in the deployed pipeline | That the adversary’s real procedure emits this event in THIS environment’s OS and logging configuration |
| 4 | Production-fired | Triggered on real adversary or red-team activity with a disposition record | Top tier |
A rule that parses is not a rule that matches. A rule that matches a test fixture is not a rule that fires in production. The gap between tier 1 and tier 3 is where silent failure lives: a rule can parse cleanly and reference valid fields while returning zero matches, because the field it keys on is empty, renamed, or now populated with different semantics than the rule was written to expect.
How do I know which of my detection rules have silently stopped working?
Four independent signals reveal silently broken rules in Splunk, Microsoft Sentinel, and Elastic Security. No single signal is sufficient alone.
Fire-rate baselining
Compare each rule’s current fire rate to its own recent baseline. Alert on a collapse to zero, or a sustained drop far below that baseline over a defined window.
Where the data lives:
| Platform | Fire-rate data | Health and execution metadata |
|---|---|---|
| Splunk | index=notable | index=_internal sourcetype=scheduler, index=_audit |
| Microsoft Sentinel | SecurityAlert, SecurityIncident tables | SentinelHealth table |
| Elastic Security | .alerts-security.alerts-* | Rules page Monitoring tab (rule execution logs, 8.x+) |
Known limitations: Fire-rate baselining fails for low-base-rate rules whose normal state is zero fires for weeks. A rule that legitimately fires twice a year cannot be watched by fire rate alone. This is why the next three signals exist.
Canary events
Inject a synthetic event known to match a rule. Confirm the rule fires end to end. This is a pattern, not a native product feature.
| Platform | Injection surface | Verification surface |
|---|---|---|
| Splunk | HTTP Event Collector (HEC) | index=notable |
| Microsoft Sentinel | Logs Ingestion API via Data Collection Rule into a custom table | SecurityAlert table |
| Elastic Security | Bulk or _doc index API into the monitored index | .alerts-security.alerts-* |
Canary events prove reach: the event survived collection, parsing, normalization, and the rule matched it. They do not prove behavior: a real adversary procedure may produce different telemetry than the canary assumed.
Periodic replay
Re-run a rule against historical data (real past events or a stored positive sample). Confirm it still alerts.
Replay must land in a tagged or non-paging sink. Without idempotency controls, validation replay creates duplicate incidents and pages the SOC.
Native support varies. Elastic Security (8.x+) has manual-run and gap-detection for detection rules. Splunk supports re-running a search over a historical time range. Microsoft Sentinel has no native analytics-rule replay.
Schema-drift alerts
Detect that the shape of incoming data changed (field renamed, removed, retyped, new enum value) before a rule silently breaks on it.
| Platform | Drift detection surface |
|---|---|
| Splunk | CIM data-model audit views in Enterprise Security, field extractions |
| Microsoft Sentinel | SentinelHealth table, data collection health monitoring workbook. A column that stops being populated shows as rising null values, not a schema error |
| Elastic Security | Mapping conflicts in Index Management, Elastic Common Schema (ECS) compliance |
Fire-rate baselining is necessary and insufficient. Use all four.
Prime Hunt’s content audit auto-maps the rules you already run to MITRE ATT&CK, and its automated threat search looks for the latest TTPs across your connected environments without moving the data.
Why do detection rules stop working?
Five mechanisms cause silent decay. In every case the rule keeps executing, returns zero matches, and execution status stays green. No error is thrown, and the coverage heatmap does not move.
1. Log source version change. The source ships a new log version and changes the semantics of an existing field. Example: an identity provider collapses an authentication outcome from three states (success, failure, challenge) to two (success, failure). A rule that detects MFA bypass by looking for “success with no preceding challenge” can no longer distinguish normal MFA flow from a bypass. The rule still runs. The field still exists. Its meaning changed.
2. Field deprecation. The source stops emitting a field. The normalization mapping now produces null for that field. A rule filtering on that field returns zero rows because every value is null. The only visible symptom: null-rate on that field climbs from near zero toward full.
3. Schema change (field renamed or retyped). The source renames a field or changes its type. The normalization mapping still points at the old name or the old type coercion. Downstream, the normalized field is empty or wrong.
Concrete example: in 2019 Microsoft added the Device prefix to the Defender ATP advanced-hunting tables (ProcessCreationEvents became DeviceProcessEvents), ahead of the unified schema that later shipped as Microsoft 365 Defender and then Defender XDR. Saved portal queries and custom detections were converted automatically; queries run through the API or stored outside the portal kept the old names and stopped returning results (Microsoft’s migration guidance). Advanced-hunting queries against a renamed table return empty, not a hard error.
4. Parser change. The source changes its log format and the parser regex no longer matches. Events fall to the dead letter queue instead of reaching the detection layer. Any rule depending on fields from those events goes quiet because the events never arrive structured.
5. Source decommissioned. A log source is retired, migrated, or has audit logging switched off. Every rule that reads that source returns zero. If a freshness SLO watches the source, detection time is within minutes. If nothing watches it, the gap is discovered on the next audit or the next incident.
The textbook case for silent decay is Windows Event 4688 (process creation). It logs when Audit Process Creation is enabled, but the Process Command Line field is populated only when a separate Group Policy setting (“Include command line in process creation events”) is also enabled. If that policy is disabled, reverts on a policy refresh, lands on a new OU, or drops on a rebuilt gold image, every detection matching on command-line content silently stops matching.
The rule still parses. Event 4688 still arrives. The field the rule keys on is empty. Nothing errors. Reproducible by toggling the policy and re-running the rule against fresh 4688 events. (Splunk Lantern: Enabling process command line logging via GPO)
All five share one trait: green means broken, not quiet. Only data-plane signals (null-rate, parse-rate, source volume, fire-rate) reveal the failure.
How do I prove my detection rules would actually fire on a real attack in my environment before one happens?
Two proof legs are required. Neither alone is sufficient.
Leg 1: emulated real procedure (the behavior leg)
Run the actual technique against the environment, not a hand-built event, and confirm the rule fires against it. This proves the rule matches real adversary behavior.
- Atomic Red Team. Per-technique atomic tests: isolated, repeatable procedures mapped to MITRE ATT&CK technique IDs. Run a single test (for example, T1059.001 PowerShell execution) and confirm the detection fires.
- MITRE Caldera. Chained adversary emulation: multiple techniques execute in sequence to simulate a campaign. Confirm correlated detections fire in order.
- Purple-team exercises. Highest-context validation. An operator executes the technique set while detection engineers observe.
Leg 2: buyer environment end to end (the reach leg)
Verify true positives in the live pipeline, with a pass record per rule. This proves the technique’s telemetry survives this estate’s collection, parsing, and normalization. A rule that works in the test lab and fails in production has demonstrated syntax, not reach.
A rule that fires on a synthetic test event has proven syntax and matching (tiers 1 and 2 on the evidence ladder). It has not proven that the adversary’s real procedure produces that event in this environment. The technique may not emit the event the rule expects on this OS version, this endpoint agent, or this logging configuration.
A rule never shown to fire against emulated behavior is presumed decorative until proven otherwise.
How do I decide which of my existing SIEM rules to retire without creating a blind spot?
Retirement is a coverage decision, not a cleanup task. Removing a rule without knowing what it uniquely covers is how blind spots form. Four inputs inform a defensible retirement, and one recorded decision closes it.
Fire history
Has the rule produced a true positive in a defined period? A rule with no true positives over the review window is a candidate, not a verdict. Some rules exist for techniques that are rare and high-impact. Check the data source health before concluding a rule is dead rather than unexercised.
Technique mapping
Which MITRE ATT&CK technique(s) does the rule cover? Without a mapping, you cannot assess whether retiring it creates a gap. If the rule is the only coverage for that technique, retirement creates a gap that must be filled before removal, or accepted and documented as a conscious risk.
Coverage overlap
Does another rule, or another data source, cover the same technique? Two rules both covering T1078 are not interchangeable if one reads identity-provider logs and the other reads endpoint telemetry. Overlap is technique-plus-data-source, not technique alone.
Data source status
Is the log source the rule depends on still active and populated? A rule against a decommissioned source produces nothing and can be retired without coverage loss.
The decision record
Detection-as-Code treats retirement as a versioned, reviewable change: a commit message, a diff, and a record, not a quiet deletion from the SIEM console. The retirement record names the rule, the technique(s) it covered, the coverage decision (covered elsewhere, accepted risk, or replaced), the date, and the author. A quarterly review cadence surfaces candidates. Prime Hunt provides a coverage and validation surface for mapping detection content to ATT&CK techniques across platforms.
Conditions that force retirement: the data source the rule depends on has been decommissioned, the technique is fully superseded by a more precise rule against the same data, or the rule produces only false positives after tuning and no redesign can fix it.
Retire stale detections. Do not suppress them. A suppressed rule inflates the rule count and creates a false sense of coverage.
What is a defensible validation cadence?
No external standard sets these frequencies. What follows is a practitioner recommendation. Each row pairs a calendar cadence with an event trigger, because decay is event-driven more than time-driven.
| Rule class | Test method | Calendar cadence | Re-validate on event | Evidence produced |
|---|---|---|---|---|
| High-volatility source (EDR, cloud audit, custom parser) | Atomic Red Team replay + fire-rate baseline | Monthly | Sensor/agent upgrade, parser change, field rename, connector change | Pass record per rule + required-field null-rate in window |
| Stable source (network, firewall) | Atomic test + fire-rate baseline | Quarterly | Source format change, ingest routing change | Pass record + source-volume-in-band |
| Correlation / stateful rule | MITRE Caldera chained emulation | Quarterly, and on any constituent-rule change | Any change to a component rule or its data source | End-to-end fire with disposition |
| Compliance-tied rule | Emulation + documented true positive | Quarterly, audit-aligned | Regulation or control change | Dated evidence bundle |
| AI-drafted rule | Full ladder (lint, unit, emulation, FP baseline) before deploy, then its class cadence | Before every deploy | Re-draft, prompt or model change | Provenance record (what drafted, who reviewed) + fire proof |
Two operational principles apply.
Fire-rate baselining, required-field null-rate monitoring, and source-volume bands are continuous. They run always rather than on a calendar, which is how you notice silent decay between scheduled validations.
Platform-native health views support this. Splunk exposes scheduler metadata in index=_internal and correlation-search execution in index=_audit. Sentinel exposes analytics-rule health in SentinelHealth. Elastic Security exposes rule execution status on the Monitoring tab (8.x+). Build the fire-rate baseline from these.
The event triggers in the table are not optional. A quarterly cadence that ignores a mid-quarter parser change misses the decay it exists to catch.
Where this does not hold
- Behavioral analytics and ML-based detections do not have discrete rule logic to replay or fixture-test. Validation for a model that scores anomalies means testing its inputs (does the expected data still arrive, in the expected shape) and checking its output distribution, not replaying a Sigma fixture.
- Threat intelligence indicator matching (hashes, IPs, domains) decays for a different reason. Indicators expire because the adversary rotates infrastructure, not because a schema changed. The validation question is freshness of the indicator feed, not whether the matching logic still parses.
- Deception-based detections (honeypots, honeytokens, canary accounts) are their own validation surface. The test is whether interaction with the deceptive asset still generates the expected alert, which is closer to canary-event injection than to rule replay.
- Schema-drift alerting is immature as a native feature. No major SIEM ships a packaged, rule-aware alert that says “a field your detection depends on just changed.” The data for drift detection exists (null-rate, mapping conflicts, CIM gaps). The wiring from field change to affected rule is manual today.
- These cadences are not standards. No regulatory body or industry framework mandates specific detection validation frequencies. The cadences above are practitioner recommendations. Adjust them to your environment’s change velocity.
DETECTION VALIDATION CHECKLIST
Pre-deploy
[ ] Rule parses against target schema (tier 1)
[ ] Rule matches known-positive fixture events (tier 2)
[ ] Rule rejects paired known-benign fixture events (tier 2)
[ ] Rule fires against emulated real procedure:
Atomic Red Team or MITRE Caldera (tier 3)
[ ] Rule fires end to end in the deployed pipeline (tier 3)
[ ] False-positive baseline documented against production telemetry
[ ] MITRE ATT&CK technique mapping recorded
Post-deploy (first 7 days)
[ ] Fire-rate baseline established
[ ] Alert volume compared to pre-deploy estimate
[ ] No unexpected false-positive volume
Ongoing
[ ] Fire-rate monitored against baseline (continuous)
[ ] Required-field null-rate monitored (continuous)
[ ] Source volume monitored for drops (continuous)
[ ] Canary event injection scheduled (per cadence table)
[ ] Emulation replay scheduled (per cadence table)
[ ] Schema-drift monitoring active on required fields
Retirement review (quarterly)
[ ] Fire history reviewed: true positives in review window
[ ] Technique mapping current
[ ] Coverage overlap assessed: another rule or source
covers the technique
[ ] Data source status confirmed: still active and populated
[ ] Retirement decision recorded with technique, coverage
rationale, date, and author
Evidence per validation cycle
[ ] Pass record per rule with date, method, and result
[ ] Null-rate and fire-rate baselines current
[ ] Schema-drift check current
[ ] Retirement log current with reason and replacement
FAQ
How do I know which of my detection rules have silently stopped working?
Four signals catch silently broken rules: fire-rate baselining against each rule’s own recent baseline, canary event injection end to end through the detection pipeline, periodic replay of stored positive samples, and schema-drift monitoring on required fields. Fire-rate baselining is the most common starting point. It fails for low-base-rate rules whose normal fire count is zero, which is why all four signals are needed together. Splunk, Microsoft Sentinel, and Elastic Security each expose fire-rate and execution health through native data surfaces described in the platform tables above. Prime Hunt provides a coverage and validation surface for mapping content to ATT&CK techniques.
How do I prove my detection rules would fire on a real attack before one happens?
Two proof legs, both required. First, run the actual technique against the environment using Atomic Red Team for per-technique tests or MITRE Caldera for chained adversary emulation, and confirm the detection fires. This is the behavior leg. Second, verify the rule fires end to end in the live pipeline with a pass record per rule. This is the reach leg. A rule that fires on a synthetic test event has proven syntax and matching. It has not proven that the adversary’s real procedure produces the same telemetry in this environment
How do I decide which SIEM rules to retire without creating a blind spot?
Four inputs make the decision defensible: the rule’s fire history over the review window, its MITRE ATT&CK technique mapping, whether another rule or data source covers the same technique, and whether the rule’s data source is still active. The retirement decision is recorded with Detection-as-Code discipline: rule name, technique covered, coverage rationale (covered elsewhere, accepted risk, or replaced), date, and author. A quarterly cadence surfaces candidates.
How often should detections be re-validated?
No external standard mandates specific detection validation frequencies. The cadence table above provides practitioner recommendations: monthly for rules against high-volatility sources (EDR, cloud audit, custom parsers), quarterly for stable sources (network, firewall), and before every deploy for AI-drafted rules. Event triggers are as important as the calendar: a parser change, sensor upgrade, or field rename triggers immediate re-validation regardless of the schedule. Fire-rate baselining and null-rate monitoring run continuously.
