The Verification Loop: Purple, Chaos and Coverage Tools

A finding that is written down is not a finding that is fixed, and a control that is declared is not a control that works. The tools of this chapter serve the activity that turns the first into the second: verification. They are the records and methods by which an organisation can say, about a technique or a control, what was tested, when, in which environment, with what result, and whether the person who says so could have been wrong in the same way as the thing being tested. They come mainly from three books: AI Purple, which defines the purple loop, its records and the independence principle (9, Nordqvist, 2026); AI Red Team, which first set out the loop from finding to verified closure and security chaos engineering (4, Nordqvist, 2026); and Architecture as Code (Chapters 9C and 13), which put codified failure experiments and drilled playbooks into the pipeline (1, Nordqvist, 2025). Smaller contributions come from AI Threat, The Transformed Security Architect, AI Blue and AI Operations.

All four kinds of tool appear. The artefacts are the purple session plan and record, the coverage record and its status rule, the regression case and the chaos experiment manifest. The methods are the purple loop itself, the practical guides for running a session and for turning a closed finding into a regression case, the steady-state hypothesis and the game day. The third-party tools are few and mostly standards or communities of practice: the Principles of Chaos Engineering, Chaos Monkey, ATT&CK and ATLAS as the vocabulary of the records, and the Center for Threat-Informed Defense's Summiting the Pyramid method. The series ships no verification software. The pipeline that evaluates a coverage status or runs a regression case is the organisation's own, and what the books provide are the records it reads and writes. Two rules recur and are applied to every tool: no tool says that something is covered without saying what was tested, when and with what result; and every tool that touches the estate says what can be undone and who can stop it, from outside the system being stopped. The authorisation tools that precede all of this, the rules-of-engagement manifest and the emulation plan, are treated in Chapter 11.

Figure 14.1 shows how the tools relate. The five stages of the purple loop produce a chain of records; the coverage record is derived from the last of them; chaos experiments enter the loop from the side, as findings if they fail and as evidence if they pass; and two things stand outside the loop and check it, the independence statement and the stop path.

Figure 14.1: The purple loop and its five stages, with the coverage record and the independence statement and stop path that stand outside it. The chaos experiment, which feeds P3 Observe and the coverage record, is described in the text.

Purple sessions

The purple loop is a way of working, and the series fixes its names. AI Purple, Chapter 4, defines five stages, P1 Target, P2 Emulate, P3 Observe, P4 Improve and P5 Verify, and says that the stages are defined by their records, not by their meetings. Three earlier books carry earlier versions of the same idea, and the chapter treats them together.

The purple loop

Kind	Method
Origin	AI Purple, Chapter 4 (five stages P1 to P5); AI Red Team, Chapter 26 (finding to verified closure); AI Blue, Chapter 11 (the blue team's side)
Purpose	Every finding ends as a verified control, detection and re-test with evidence; the loop returns closed findings as regression cases
Use when	Always, as the frame for any exercise, campaign or experiment whose results must lead to change
Inputs	Intelligence, the threat register, incidents, the scenario library
Outputs	A chain of records: scenario, emulation plan, finding and session record, change record, regression case and evidence manifest
Identifier family	SCN-*, EMU-*, FND-*, PRP-*, CHG-*, REG-*, EVD-*

AI Red Team began the loop at the finding; AI Purple adds two stages in front, because the quality of a finding depends on what was chosen and how it was run. The distinction between AI Blue's view and AI Purple's is one of emphasis: the blue team's version turns an exercise list into four kinds of change (a new detection, a finding against triage, a finding against a playbook or rota, a defensive technique the team had but did not use) and leaves behind a regression test, whereas the purple version defines the whole chain. AI Purple also fixes the closure rule: a finding is closed only when the re-test and at least one variant pass, the evidence is attached, a named person signs and the closure is dated. It names five finding states: open, implemented but unverified, re-test failed, closed and verified, and risk accepted, of which only one is closed (9, Nordqvist, 2026).

The purple loop inventory (Appendix B.1 of AI Purple) is the starting artefact: it maps who performs P1 to P5 today, red, blue or contracted, with which tools, and which parts are automated. Its output is a stage map, loop/stage-map.yaml, from which a delegation profile can be written. The book describes the inventory but gives no full template.

Purple session plan (PS-*)

Kind	Artefact (with the method "Running a quarterly purple-team session")
Origin	AI Red Team, Chapter 26
Purpose	A quarterly purple-session plan with chair, agenda, replay, closure criteria and works-council limits
Use when	Planning the periodic session at which red, blue and the white team read results together
Inputs	Open findings from exercises and chaos experiments, the rules of engagement
Outputs	Findings with owners, closing actions, re-test scenarios and dates; replays added to the regression catalogue
Identifier family	PS-*
Full template	AI Red Team, Chapter 26 (the PurpleSession kind)

The plan fixes the chair (the white-team lead), an agenda limit (at most ten findings, medium severity or above), the replay rules (catalogued benign scenarios only, a stop path named, one scenario with the AI unavailable), the closure rule (re-test passed, evidence attached, owner accepted) and the works-council limit on analyst metrics (aggregate only). The defaults are one session per quarter and one within ten working days of a major incident or a failed high-severity chaos experiment, with re-tests within 30 days for high-severity findings and 90 days for medium. A tool named purple-align appears in the Hansa Mutual example as a read-only reconciliation at Mode 4 and Level 0; the book never defines it, and it is not treated as a tool here (4, Nordqvist, 2026).

Purple session record (PRP-*) and the session method

Kind	Artefact (with the method "Purple session: blameless, expectations first")
Origin	AI Purple, Chapter 9 and Appendix A ("Purple session record PRP-NR-2027-02"); Appendix B.7
Purpose	The record of a session in which red and blue read the result together: expectations, outcomes, decisions and findings raised
Use when	After every emulated scenario or campaign
Inputs	The emulation plan, the sealed expectation sheet, the tagged log, the detection and case records
Outputs	One outcome per step in the seven-class vocabulary of Chapter 11, findings raised, disagreements and how they were resolved
Identifier family	PRP-NR-*, PRP-HM-*
Full template	AI Purple, Appendix A

The distinguishing feature of the method is that the expectation is written, hashed and sealed before the run, so that the session compares a result with a commitment, not with a memory. In the Nordvik record the six steps of a spray-then-success scenario end as prevented, detected, alerted but not triaged, missed, no telemetry and not tested; the second of these, "alerted, closed by the provider's automation", is the finding that matters most. The session is blameless and aggregate-only: the charter below says what is not recorded (9, Nordqvist, 2026).

Purple session charter, exercise-information notice and exercise decision-rights table

Tool	Kind	Origin	One-line purpose
Purple session charter (PSC-*)	Artefact	AI Purple, Chapter 23 and Appendix A (PSC-NR-2028-01)	Blameless ground rules: who attends, what is recorded (outcome per step by system and class), what is not (audio or video, the name of the analyst on shift against an outcome, per-person timings) and who may read it
Exercise-information notice (EIN-*)	Artefact	AI Purple, Chapter 23 and Appendix A (EIN-NR-2027-002)	The notice to staff representatives before an exercise that exposes staff, stating what will happen, what it tests and that results concern the control and are reported in aggregate
Exercise decision-rights table (DRF-*)	Artefact	AI Purple, Chapter 20 and Appendix B.20	Names who may authorise, change scope, stop, accept a residual finding and declare a technique covered, and how a real incident during an exercise is handled; output is the table and a tabletop record (TTX-*)
Sector information-sharing protocol (SHP-*)	Artefact	AI Purple, Chapter 22 and Appendix A (SHP-HM-2028-01)	Sharing exercise results across sector partners without exposing sensitive detail, using the Traffic Light Protocol
Tabletop on the OpenAI and Hugging Face incident timeline	Method	AI Governance, Chapter 19	Using the timeline of a published incident to test the organisation's own decisions
Half-day exercise assessment	Method	AI Blue, Chapter 22	Evidence in half a day of what a team can do unassisted and with its assistants, graded by two assessors, one from outside the person's line, and written as competence records (CMP-*); employee representatives receive the schema and rubric in advance
Resilience exercise design record (NR-AWR-*) and "Designing a deepfake-resilience exercise"	Artefact and method	AI Red Team, Chapter 15	Testing the process and not the person, with consent, scope and success criteria written first

The first three are the people-facing counterpart of the manifest of Chapter 11. A purple session is an exercise in which staff are observed, and the series treats the limits on that observation as part of the tool: the charter forbids per-person timings, the notice reaches representatives before the window, and the decision-rights table says who may declare a technique covered. The last two rows belong in part to Chapter 21, which treats competence; they are listed in data/findings/chapter_14.md for the editor's decision.

Coverage and evidence age

AI Purple, Chapter 17, replaces the technique heat map with a record that carries age. It defines "covered" for one condition only: a person can show, from records, that a specified technique was exercised in a specified environment inside a window of time, that the defence produced a specified result at a specified layer, and that the evidence has not expired. The tools below implement that definition. The vocabulary of layers is the outcome taxonomy of Chapter 11, and the technique identifiers are ATT&CK or, for AI systems, ATLAS (67, MITRE, n.d.) (68, MITRE, n.d.).

Coverage record (COV-*)

Kind	Artefact
Origin	AI Purple, Chapter 17 and Appendix A ("Coverage record COV-HM-0112"); Appendix B.17
Purpose	Per technique and environment: what was tested, when, how, with what outcome and when the evidence expires
Use when	Whenever someone wishes to claim that a technique is covered
Inputs	Regression cases, evidence manifests, assertions, the independence statement
Outputs	A record whose status is computed from its fields and never edited by hand
Identifier family	COV-NR-*, COV-HM-*
Full template	AI Purple, Appendix A (quoted below)

The record is derived. It is rebuilt whenever a regression case, an evidence manifest or an assertion changes. It names the technique and the knowledge-base version, the environment and tier, what was tested (scenario, regression case, variants, whether one was held out, the safety class and the runner), the date of the last run, the result by layer, the evidence manifest, an expiry with a clock and conditions that end the evidence earlier, the independence statement that applies and the person who declares. The status is one of six values: verified, expired, implemented-unverified, failed, asserted and untested. Only verified counts as covered. An expired pass is reported as not tested, with the date of the last pass, "the only statement that remains true" (9, Nordqvist, 2026).

# Illustrative: Hansa Mutual, coverage record
kind: CoverageRecord
id: COV-HM-0112
technique: {atlas: AML.T0053, note: family of SCN-HM-0058, knowledge_base: ATLAS-current-at-signature}
environment: claims-triage-agent/production-replay
tier: 1
tested:
  scenario: SCN-HM-0058                     # authored by the group red team
  regression_case: REG-HM-0112
  variants: [{id: v1-other-market, held_out: false}, {id: h1-tool-chain, held_out: true}]
  safety_class: S1
  run_by: HM-SEC-004                        # MND-HM-EMU-06, Level 3 for S1 regression cases
last_tested: 2028-02-09
result:
  prevented: true
  detected: {by: DET-HM-0561, minutes: 4, triaged_by: person}
  weakest_layer: none
  findings: [FND-HM-2027-031]               # closed and verified
evidence: {manifest: EVD-HM-2028-0121, signed: true, reproduced_by: person}
expiry:
  clock_days: 90
  expires: 2028-05-09
  conditions: [model_change, prompt_template_change, manifest_change, DET-HM-0561_edit]
independence: {statement: IND-HM-2028-Q1, valid_to: 2028-05-22, drafter_checker_family: separate}
status: verified                            # computed from the fields above, never edited
declared_by: lena-vogel                     # head of detection engineering
declared_on: 2028-02-12

Coverage-status rule file

Kind	Artefact
Origin	AI Purple, Chapter 17 and Appendix A
Purpose	The ordered rule that turns the fields of a coverage record into a status
Use when	In the pipeline that evaluates coverage, on every change to a regression case, evidence manifest or assertion
Full template	AI Purple, Appendix A (quoted below)

The first rule that matches wins, and no override is allowed; an extension is a recorded decision with a named person, a reason and a new date.

# Illustrative: status rule, evaluated by the pipeline on every change to a REG, EVD or assertion
statusRule: coverage-status v1.0.0
order:                                   # first match wins
  - {when: no_record_or_no_run,                       status: untested}
  - {when: last_result == fail,                       status: failed}
  - {when: finding_open and change_merged,            status: implemented-unverified}
  - {when: tested_by == assertion_only,               status: asserted}
  - {when: age_days > clock_days or condition_changed, status: expired}
  - {when: pass and variants_run and negative_control, status: verified}
  - {when: otherwise,                                    status: failed}
reportAs:
  verified_count: all records with status verified
  of_which_independence_shown: verified records whose independence field reads shown
  covered: [verified]
  notCovered: [expired, implemented-unverified, failed, asserted, untested]
override: {allowed: false, extension: "recorded decision, named person, reason, new date"}

Coverage page, CSF profile as code and the methods of coverage

Tool	Kind	Origin	One-line purpose
Coverage page (PRI-NR-2028-01)	Artefact	AI Purple, Chapters 17 and 18, Appendix A	A computed page per priority list, with counts, an age distribution, the oldest unexpired and oldest expired claim, and a named declarer
CSF profile kept as code	Artefact	AI Purple, Chapters 3 and 17; introduced in AI Red Team, Chapter 3	A NIST CSF 2.0 profile that refers to coverage-record identifiers and computed status by reference, instead of stating maturity by assertion (53, NIST, 2024)
Coverage as a claim (status, evidence age, expiry, honest "not tested")	Method	AI Purple, Chapter 17	Replace heat maps with records carrying age and expiry
Heat map replaced by coverage records	Method	AI Purple, Chapter 17 (Practical guide)	Convert a technique heat map into coverage records, and retire the colour map for reporting
Summiting the Pyramid	Third-party tool (method)	AI Purple, Chapter 17	The Center for Threat-Informed Defense method that separates implementation coverage from detection quality, used to qualify what "visibility" means in a coverage record

The Nordvik page as of 26 April 2028, as the book prints it (illustrative), reports nine techniques as five tested, two asserted by the provider and two not tested, in three states and not in a colour. Hansa's conversion of its 34-technique heat map reproduced the board's figure of 25 verified and showed that four of the nine cells that now read as not verified had been green on the old map, because their last result was a pass and the map had no age (9, Nordqvist, 2026). That conversion is the practical case for the method. Summiting the Pyramid is described by its project page as evaluating the robustness and precision of the signals behind detection visibility (91, Center for Threat-Informed Defense, 2026); the series uses it only to make "detected" more specific.

Control-to-threat coverage matrix (CCM-*)

Kind	Artefact
Origin	AI Threat, Chapter 30 (worked example CCM-HM-2027-01)
Purpose	Maps registered threats to controls and states whether each control is verified, declared or absent
Use when	Reporting to a board how well the controls against a threat register hold
Inputs	The threat register, control identifiers (CTRL-*), test records
Outputs	A matrix per assessment, a rule (observable or expected threats need a verified preventive and detective control) and a list of findings with owner and due date
Identifier family	CCM-*, CTRL-*
Full template	AI Threat, Chapter 30

The matrix is the register-level cousin of the coverage record. Each control is classified verified, declared or failed according to whether a test that re-runs its scenario has passed recently, and each finding becomes the next step of the purple loop. Its classes map onto the coverage statuses once, in the table of AI Purple, Chapter 17 (5, Nordqvist, 2026) (9, Nordqvist, 2026).

Board page for the blue team and the closure-rate replacement

Tool	Kind	Origin	One-line purpose
Board page (reporting/blue-team-page.yaml, BTP-*)	Artefact	AI Blue, Chapter 20, Appendix A and Appendix B.18	One page of speed, coverage, judgement and learning, reported by team and never by person, with a gaming check for each measure and at most three decisions for the board
Replacing a closure-rate target with three measures and a gaming check	Method	AI Blue, Chapter 20	Measure speed on the right population, and attach to every measure the check that shows it moving for the wrong reason

The Hansa example reports coverage as 34 priority techniques, 25 verified, 7 declared and 2 failed, with the oldest passing test 74 days old; the gaming check is the date of the last passing test (8, Nordqvist, 2026). The reconciliation of the coverage vocabulary of AI Blue (verified, declared, failed) with that of AI Purple (six statuses) is given in Chapter 17 of the latter.

Regression cases

Regression case (REG-*)

Kind	Artefact (with the method "Turning a closed finding into a regression case")
Origin	AI Purple, Chapter 11 and Appendix A ("Regression case REG-NR-0004"); Appendix B.10
Purpose	A closed finding kept as a repeatable test with variants, a negative control and a cadence
Use when	At P5 Verify, when the exact re-test of a finding has passed
Inputs	The finding, the exact scenario, authors of variants, the owner
Outputs	A case that runs on the stated triggers; an evidence manifest (EVD-*), signed by a named person
Identifier family	REG-NR-*, REG-HM-*
Full template	AI Purple, Appendix A (quoted below)

Re-testing only the exact scenario protects the path and not the technique. The case therefore carries the base scenario, variants in path, context, rate or shape, a held-out variant written by someone other than the author of the fix, and a negative control, a benign activity that resembles the technique and must not alert. The pass criterion is stated at a layer; in the Nordvik example an alert that is closed by the provider's AI fails.

# Illustrative: Nordvik Region, regression case for FND-NR-2027-011
kind: RegressionCase
id: REG-NR-0004
finding: FND-NR-2027-011
source: autumn-2027-catalogue-run-PRP-NR-2027-02
technique: {attack: T1110.003, followedBy: T1078}
protects: [DET-NR-0001, MS-1/runbook-low-severity-closure]
owner: ROLE-NR-SEC-DET
environment: benefits-triage-signin      # production, safety class S1, injected record
passCriterion:
  layer: triaged-by-person               # an alert that is closed by the provider's AI fails
  maxMinutes: {inHours: 15, outOfHours: 30}
scenarios:
  - {id: base,    kind: exact,    ref: SCN-NR-0001}   # S1 injected-record form; the live form is S2
  - {id: v1-rate, kind: variant,  change: rate-below-threshold,   author: model-proposed-person-reviewed}
  - {id: v2-zone, kind: variant,  change: source-network-class,   author: karin-lund}
  - {id: v3-hours, kind: variant, change: out-of-hours-delivery,  author: karin-lund}
  - {id: h1,      kind: held-out, author: elin-berglund, visibleToFixAuthor: false}
negativeControl: {id: n1, case: bulk-password-resets-after-holiday, mustAlert: false}
safety: {class: S1, cleanup: remove-injected-record, stopPath: [mssp-shift-lead, nr-duty-analyst]}
triggers: {onChange: [DET-NR-0001, MS-1, provider-ai-config], schedule: quarterly-MS-2}
evidenceExpiryDays: 180

The practical guide has nine steps: restate the hypothesis; run the exact scenario and stop if it fails; write at least three variants for a priority technique and one for any other, with a person reviewing any AI-proposed variant; have someone who did not write the fix author a held-out variant; add a negative control; confirm safety class, tag, cleanup and stop-path holders for every scenario; run the full case once and sign the evidence; set the triggers; set the expiry. The defaults are a re-run at least every 30 days and on change with a 90-day expiry for priority techniques, and every 90 days with a 180-day expiry for others (9, Nordqvist, 2026).

The method "Drafting rules of engagement for an AI exercise" (AI Red Team, Chapter 7) was assigned to this chapter by the inventory but belongs with the manifest in Chapter 11, where its output is treated.

Independence between tester and tested

The questions this chapter was set include how independence is kept. No inventory entry carries it as a tool, because AI Purple, Chapter 15, treats it as a principle with controls; but every tool above depends on it, so the chapter states it. The principle is that a check is evidence only if the side that performs it can fail in ways the side being checked does not share. A re-test that asks the system that produced a fix whether the fix works is not a check, and a coverage report that an emulation agent and a detection agent have agreed between themselves is not a measurement. AI Purple separates five kinds of independence, each with its own shared dependency and its own proof (9, Nordqvist, 2026):

Kind	Shared dependency that hides failure	Control
Model	The same model family behind tester and tested, or drafter and critic	A critic on a different family, used for questions and not verdicts
Data	The same training data, retrieval corpus or threat-intelligence feed	A held-out set built from sources the detection side does not read
Authorship	The same author, human or agent, for scenario and detection	Different authors; human-authored novelty in every cycle
Vendor	The same vendor or gateway tenant, including provider AI	Disclosure, notice of change, a second supplier or human route for the cells that matter
Time	The same moment, such as a re-test run straight after the fix	Re-test after a delay and after changes, with an expiry on the evidence

Two tools implement the principle. The held-out scenario is written by a person, kept where the detection side and its agents cannot read it, run on a schedule the detection side does not control and burned on use, moving to the open library as a regression case; its default share is at least 15 per cent of the library and not fewer than eight scenarios. The seeded gap is a known weakness introduced on purpose, under the rules of engagement and a change record, in a place where it can be undone, to test whether the loop itself finds it; the catch rate has a stop rule, below which delegation in that class is frozen until a person has reviewed why. The method of the architect's AI team, "Second agent as adversary" (The Transformed Security Architect, Chapter 14), is the same idea at the level of a single proposal: the critic never approves, and a person disposes of each objection (6, Nordqvist, 2026). This chapter's own conclusion, in the Summary, is that the five kinds are not ranked: a report should say which of them it shows.

Security chaos engineering

Security chaos engineering applies a method from reliability engineering to security controls. The distinction drawn by AI Red Team, Chapter 25, keeps the tools apart: penetration testing and red teaming look for paths an adversary could use, in a bounded window; breach-and-attack simulation replays catalogued techniques as regression testing; security chaos engineering injects conditions, including failures of security controls themselves, and checks steady-state hypotheses across every Function of the NIST CSF (4, Nordqvist, 2026). The third-party sources are the practice's own literature.

Principles of Chaos Engineering and Chaos Monkey

Kind	Third-party tools (a published set of principles; open-source software)
Origin	Principles: AI Red Team, Chapter 25; The Transformed Security Architect, Chapters 10 and 11; AI Purple, Chapters 8, 11 and 15; AI Operations, Chapters 1, 9, 10, 12, 17 and 19. Chaos Monkey: Architecture as Code (Chapter 13); AI Red Team, Chapter 25
Purpose	The principles state the method; Chaos Monkey is the tool from whose success at Netflix the practice grew
Use when	Designing any experiment that tests a system under turbulent conditions

The Principles of Chaos Engineering community condense the method into building a hypothesis around measurable steady-state behaviour, varying real-world events, running experiments in production, automating them to run continuously and minimising blast radius (90, Principles of Chaos Engineering community, n.d.). Basiri and colleagues described chaos engineering as experimenting on a distributed system to build confidence in its ability to withstand turbulent conditions in production (216, Basiri et al., 2016). Netflix's Chaos Monkey is open-source software that randomly terminates instances in production (209, Netflix, n.d.). Rinehart and Shortridge adapted the practice to security as proactive experimentation to identify failures in security controls (217, Shortridge & Rinehart, 2023). Architecture as Code (Chapter 8) also names Gremlin and Chaos Mesh as chaos engineering platforms, in passing and without describing them; the series claims nothing more about them. Four inventory entries for the Principles and two for Chaos Monkey recur across the books and are treated here as two entries.

Steady-state hypothesis for security

Kind	Method
Origin	AI Red Team, Chapter 25
Purpose	A falsifiable statement of condition, expected outcome and observation, measured from outside the control tested
Use when	Before any security chaos experiment

"The EDR works" is not a hypothesis. The book's example names the condition (a benign emulation of credential dumping, ATT&CK T1003, on a managed workstation), the outcome (an alert of severity high or above in the SOC queue within five minutes, triaged within fifteen) and the observer. Security hypotheses differ from reliability hypotheses in four ways: the expected outcome is often an action by people, so an alert never triaged is a failed experiment; absence is a result, as with a silent log source; the adversary adapts, so experiments must vary; and every experiment produces a record for the evidence manifest (4, Nordqvist, 2026).

Security chaos experiment manifest (SCE-*)

Kind	Artefact
Origin	AI Red Team, Chapter 25 and Appendix A (25_CODE_CHAOSEXP); The Transformed Security Architect, Chapter 11 (the recovery experiment SCE-RC-007)
Purpose	A catalogue entry that injects a condition to verify a control, with rollback, blast radius and failure routing
Use when	Verifying one control property continuously
Inputs	A reviewed catalogue entry, the safety-boundary policy, the standing rules of engagement
Outputs	A result against a control identifier; on failure, a finding routed to the purple backlog
Identifier family	SCE-* (for example SCE-DE-014, SCE-RC-007)
Full template	AI Red Team, Appendix A (quoted below)

The entry injects a condition, not an attacker technique: here, that log forwarding from one host stops, and the hypothesis is that the SOC queue shows a "log source silent" alert within 30 minutes. The injection is a named actuator with a target selector, the rollback is automatic and verified, the blast radius is one target and excludes domain controllers and identity providers, and autonomy is promoted from test to production after 20 safe runs.

# Illustrative: catalogue entry for a Detect experiment
kind: ChaosExperiment
id: SCE-DE-014
title: Silent log source is noticed by the SOC
csf: [DE.CM]
owner: detection-engineering
hypothesis:
  condition: log forwarding from one host in scope stops
  expected: alert "log source silent" in SOC queue
  withinMinutes: 30
injection:
  actuator: log-forwarder-pause
  targetSelector: {tier: [test, production], role: [app-server]}
rollback: {action: resume-forwarder, verify: events-received-again, automatic: true}
blastRadius:
  maxTargets: 1
  excludedRoles: [domain-controller, identity-provider]
autonomy: {test: 3, production: 3}     # promoted after 20 safe runs in test
evidence: {controlIds: [CTRL-SOC-021]}
onFailure: {createFinding: true, route: purple-backlog}

The recovery experiment of The Transformed Security Architect (SCE-RC-007, illustrative, platform HM-SEC-006) uses the same kind with a different condition: a running instance replaced by an altered artefact not in the evidence bundle, with four checks (integrity check rejects the artefact within 10 minutes, degraded mode active within 15, redeployed and re-evaluated within 90, no pending action replayed). Its guide, "Designing a first security chaos experiment", adds that the hypothesis must include the human step (6, Nordqvist, 2026).

Chaos safety-boundary policy

Kind	Artefact
Origin	AI Red Team, Chapter 25 and Appendix A (25_CODE_CHAOSPOLICY)
Purpose	A policy bounding blast radius, data, timing, autonomy and abort for AI-planned chaos experiments
Use when	Before a chaos platform runs any experiment, as the standing rule its policy decision point evaluates
Identifier family	HM-SEC-*-boundaries (the example is HM-SEC-006-boundaries)
Full template	AI Red Team, Appendix A

The policy separates planning from execution (the planner holds no estate credentials, the actuator credentials are issued just in time per experiment), allows only reviewed catalogue entries and forbids the AI to generate new injections, sets maximum autonomy by environment (Level 3 in production only for types with 20 safe runs, Level 2 for new types, destructive actions forbidden), restricts data to synthetic accounts, fixes working-hours windows and blackouts, requires an abort that is independent of the planner and names an out-of-band stop path tested by its own experiment. The platform is a tier-0 asset in scope of the threat-led exercise. The design draws on the July 2026 incident, in which agents operating in a security-testing context exceeded their mandate (4, Nordqvist, 2026).

Methods and records of chaos practice

Tool	Kind	Origin	One-line purpose
Introducing security chaos engineering in stages	Method	AI Red Team, Chapter 25 (Practical guide)	Start with ten to fifteen catalogue types, at least one per CSF Function, at Level 2 in test with the SOC informed; add the sealed ledger, then the planner, then production one type at a time; include the platform in the next threat-led exercise
Security chaos experiment	Method	The Transformed Security Architect, Chapter 11	Verify one recoverability property safely and move a control from declared to verified
Chaos experiment record	Artefact	The Transformed Security Architect, Chapters 11, 25 and 26	Experiment with hypothesis, safety boundaries, abort holder and result (the SCE-* kind again)
Game day and chaos experiment	Method	AI Operations, Chapter 19	An experiment record with five fields, refused if any is missing: hypothesis, scope and blast radius, abort criteria, a stop path outside the system under test and the runner, and a result measured by an independent probe
Security chaos engineering with Chaos Monkey experiments	Method	Architecture as Code (Chapters 9C and 13)	Failure experiments described next to the infrastructure code, in version control, run in lower environments first, with concurrent injection limited and windows broadcast so on-call staff know which alerts belong to the exercise
Playbooks as state machines, exercised continuously	Method	Architecture as Code (Chapter 9C)	Express response actions as workflows with guard conditions, rollback and evidence capture, and exercise them by scheduled drills
Game day with the agents off (dark day) and dark-day record (DKD-*)	Method and artefact	AI Operations, Chapters 19, 26 and 29	A scheduled period in which operators run a defined workload without the operations agents; only the actors are switched off, never the observer or the stop path; a missing step becomes a change to a runbook; the record has no separate template

Two lessons run through the table. First, the experiments of AI Red Team are conditions injected into controls, while the experiments of AI Operations are conditions injected into services (10, Nordqvist, 2026); both use the same five elements and the same rule that the stop path sits outside the system being stopped. Second, the sealed ledger resolves a tension that every programme meets: announced experiments no longer verify detection, unannounced ones waste analysts' time, and so the ledger seals the entry from the SOC for the detection window and lets the duty shift lead ask "is this an experiment?" without seeing the schedule (4, Nordqvist, 2026).

Other entries assigned to this chapter

Entry	Origin	Disposition
ADR dashboards (lifecycle timeline, policy coverage heat map, review debt tracker)	Architecture as Code (Chapter 4)	Architecture decision metadata, not verification; to Chapter 4
Architecture coverage script and ci/validate-architecture.py	Architecture as Code (Chapter 6)	Verifies architecture workspace rules; to Chapter 4
Interview exercise testing judgement with and without AI	The Transformed Security Architect, Chapter 22	Competence; to Chapter 21

Across the usage modes

Verification is where the independence principle meets the usage modes. A Mode 5 agent that runs a scenario does not also grade it, and the autonomy ceilings for the loop are those of AI Purple, Chapter 4: Level 2 Supervised for P1 and P4, Level 3 Bounded for catalogued S1 scenarios at P2 and for running and recording a re-test at P5, closure always a named person's signature, and Level 4 Autonomous never for an emulation action that is irreversible, leaves the signed scope, touches personal data beyond what the exercise may touch or targets a named person. For chaos experiments the ceilings are Level 2 for new types and Level 3 for proven ones, with Level 4 not used (4, Nordqvist, 2026) (9, Nordqvist, 2026).

Mode	Role in verification	Limit
Mode 0 Ambient AI	The provider's or product's AI closes or ranks alerts that an exercise then reveals	Surfaces as "alerted but not triaged"; handled by contract terms
Mode 1 Conversational assistants	An analyst drafts queries over exercise logs or hypotheses for the catalogue	No tagged data to an unapproved service; review before use
Mode 2 Embedded workplace assistants	Summaries of session notes or experiment results	A summary is never the record; it can hide a partial failure
Mode 3 Agentic work tools	Tables of outcomes from saved queries; building actuators	A person confirms each classification; the engineer's credentials must not touch the estate
Mode 4 Built AI applications	A correlator joins emulation, detection and case records; the chaos planner and evaluator	Read-only, structured fields, no estate credentials; a person signs
Mode 5 Autonomous production agents	Runs catalogued S1 scenarios and re-tests; avoided in chaos platforms, where only deterministic actuators act	Does not grade its own run; stop path outside the agent

Nordvik Region and Hansa Mutual

Nordvik Region buys the loop. Its coverage page lists nine techniques for the benefits-triage service in three states (tested, asserted by the provider, not tested); its regression case REG-NR-0004 makes the managed provider's closure of an alert by automation a failure; its chaos programme NR-SEC-004 runs a catalogue of twelve experiment types weekly in test environments, with the managed provider's shift lead holding the stop path out of hours and production limited to the honeytoken path and restore drills in an isolated copy. Karin Lund declares the coverage page, and the contracted tester reproduces the verified rows, which gives the page independence by role.

Hansa Mutual runs the loop. Its pipeline HM-SEC-004 produces regression cases and coverage records at a rate no one reads by hand; its chaos platform HM-SEC-006 runs proven catalogued types in production at Level 3 Bounded within working-hours windows, with works-council agreement where an experiment touches approval flows or endpoint telemetry. In March 2028 it converted the heat map behind its board page to coverage records and found that four of the nine cells now reading as not verified had been green on the old map. Its independence statement IND-HM-2028-Q1 covers seven of the 25 verified techniques, so the board figure carries that qualifier. The difference is that Nordvik's risk is a page accurate on the day it is declared and then not refreshed, and Hansa's is a page refreshed so often that nobody reads its age; both guard against it by reporting evidence age.

Treatment of the slice

All 51 inventory entries of the slice are accounted for. Three are reassigned: the two architecture entries to Chapter 4 and the interview exercise to Chapter 21. Two, purple-align and the Gremlin and Chaos Mesh platforms, are named only in passing, as the originating books name them. The other 46 are treated in the entries and family tables above, merged where the same tool recurs in several books (the three versions of the purple loop, and the Principles of Chaos Engineering and Chaos Monkey). Appendix A holds every entry with its originating location.

Summary

Verification is the activity that gives the other tools of the book their meaning, and its tools share a design. They are records, not meetings: the session record, the coverage record, the regression case and the chaos experiment each have an owner, a date and a state, and the status of a coverage record is computed from its fields and never edited. They carry age, because evidence expires: a pass without a date and a condition is the same as an assertion. They are checked from outside, because a check is evidence only if the side that performs it can fail in ways the side being checked does not share, and the five kinds of independence (model, data, authorship, vendor and time) are not interchangeable. They are safe by construction: every tool that touches the estate names a rollback, a blast radius and a stop path held outside the system it stops, and the chaos platform that injects the conditions is itself a tier-0 asset that is tested. And they end in a person: closure is a named person's signature, never an agent's. The practical consequence for the reader is that the question to ask of any claim of coverage is the one the series asks: what was tested, when, in which environment, with what result, and by whom, other than the thing being tested.

Sources

1. Nordqvist, G. Architecture as Code. (1, Nordqvist, 2025)
2. Nordqvist, G. AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (4, Nordqvist, 2026)
3. Nordqvist, G. AI Threat: Are the Dystopias Coming True? Evidence, Risk and Response for European Enterprises and Public Authorities. (5, Nordqvist, 2026)
4. Nordqvist, G. The Transformed Security Architect: Role, Craft and Career in the Age of AI in European Enterprises and Public Authorities. (6, Nordqvist, 2026)
5. Nordqvist, G. AI Blue: How Artificial Intelligence Transforms the Blue Team in European Enterprises and Public Authorities. (8, Nordqvist, 2026)
6. Nordqvist, G. AI Purple: How Red and Blue Learn Together When AI Joins the Loop in European Enterprises and Public Authorities. (9, Nordqvist, 2026)
7. Nordqvist, G. AI Operations: How Artificial Intelligence Transforms IT Operations in European Enterprises and Public Authorities. (10, Nordqvist, 2026)
8. NIST. The NIST Cybersecurity Framework (CSF) 2.0. (53, NIST, 2024)
9. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (67, MITRE, n.d.)
10. MITRE Corporation. MITRE ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. (68, MITRE, n.d.)
11. Principles of Chaos Engineering community. Principles of Chaos Engineering. (90, Principles of Chaos Engineering community, n.d.)
12. Center for Threat-Informed Defense. Summiting the Pyramid. (91, Center for Threat-Informed Defense, 2026)
13. Netflix. Chaos Monkey. (209, Netflix, n.d.)
14. Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., Rosenthal, C. Chaos Engineering. (216, Basiri et al., 2016)
15. Shortridge, K. Security Chaos Engineering: Sustaining Resilience in Software and Systems. (217, Shortridge & Rinehart, 2023)