Change, Restore and Runbook Tools
When a service must be altered or brought back, the operator acts, and the tools of this chapter decide how far an action may go before somebody else looks. They serve the activities that AI Operations calls O3 Change and O4 Restore (10, Nordqvist, 2026): classifying and recording a change, releasing it in stages, reconciling declared with actual state, writing a procedure that a machine may run, and fixing the calendar and the windows within which any of this happens. They are the series' instruments for one rule: what can be undone, and who can stop it, outside the system being stopped.
The four kinds of tool occur. The artefacts are the change record and change class (CHG-*), the progressive-delivery policy (ROL-*), the runbook manifest (RBK-*), the model-change record, the freeze calendar (FRZ-*), the catalogue of first remediation actions, the release record and the drift runbook of AI Platform and Security, and two listings of Architecture as Code. The third-party tools are the GitOps and progressive-delivery engines (Argo CD, Flux, Argo Rollouts, Flagger, Kargo), the OpenGitOps principles, the service mesh Istio and two workflow engines (Argo Workflows, Temporal). The methods are progressive delivery, feature flags, rollback safety, freezes, remediation tiers, safe execution of runbooks, signed runbooks and remediation windows. Most come from AI Operations (Book 10), which defines the change and restore artefacts; Architecture as Code (Book 1) and AI Platform and Security (Book 3) supply the earlier forms; AI Red Team (Book 4) and AI Purple (Book 9) each supply one.
Figure 16.1: From change to restore, with the stop path outside the platform.
The chapter follows the figure. Change classes and records come first, then progressive delivery and GitOps, then runbooks and remediation, then the tools that fix when change may happen. The third-party tools are described only as far as their documentation supports and the series claims; the versions current at the time of the originating book are not repeated, because they age quickly, and AI Operations, Chapter 8, dates its own snapshot.
Change classes and records
A change is classified by the diff and the target, not by what its author says. That sentence is the core of this family. A declared class is a claim, the computed class is the fact, and the tools below make the computed class the one that the pipeline acts on.
Change class and change record
| Field | Content |
|---|---|
| Kind | Artefact |
| Origin | AI Operations, Chapters 8 and 13 and Appendices A.4 and B.6 |
| Purpose | A change class states what may be released without a person in the path; the record carries author, reconciler, computed class, plan summary, independent approvers, tested rollback and verification |
| Use when | For every production change; the class file is written once per kind of change that is to be released without per-instance approval |
| Inputs | The diff, the service record, the freeze calendar, the approvers, a probe that the pipeline does not run |
| Outputs | A change record in version control; a gate decision |
| Identifier family | CHG-NR-*, CHG-HM-* for change records; HM-STD-* for classes |
| Full template | AI Operations, Appendix A.4 (CHG-NR-2028-0007, class HM-STD-002 and an illustrative Rego gate) |
The change record names the author as a person or an agent, and for an agent it records the tool, the identity, the inputs and what was not tested. It also names the reconciler, the engine that applied the change, because without it a person who was not there cannot reconstruct who applied what. The excerpt below is CHG-NR-2028-0007, an AI-authored infrastructure change at Nordvik Region, with the fields on the author's inputs, the policy results, the plan summary and the stated limit of the claim removed. It is quoted from the version of Book 10 in the repository.
change_record:
id: CHG-NR-2028-0007
service: SVC-NR-002 # Tier 1
title: Automatic renewal of the portal certificate at the recovery site
author:
kind: agent
tool: agentic-work-tool # Mode 3, platform team permission
identity: wl-nr-platform-draft
provenance_stored: references_only # no typed prompts (MBL-NR-IT-2027-01)
reconciler:
mode: push # applied by the pipeline credential; no continuous reconciler
engine: terraform-pipeline
identity: wl-nr-pipeline-apply
sync_policy: "apply on merge inside the window; no automatic re-apply; drift detector read-only"
declared_class: normal
computed_class: normal # Tier 1 service, no catalogue match
approvers:
- {id: maja.lindqvist, kind: person, team: platform}
- {id: oskar.berg, kind: person, team: infrastructure}
window: "2028-10-17 20:00-22:00"
rollback:
plan: "revert commit, destroy 3 added resources, no data touched"
reversibility: full
tested_on: 2028-10-12
result: passed
stop_path: "duty manager and platform engineer on call, pipeline kill switch"
verification:
probe: external-tls-probe # run by monitoring, not by the pipeline
result: "chain valid, renewal object present, 89 days to expiry"
checked_at: 2028-10-17T21:12
The change class is the other half. It turns a kind of change that people have made by hand many times without a correlated incident into data that a gate can evaluate. The class below is quoted in full from the same appendix; the book's practical guide (below) gives the steps that lead to it.
change_classes:
- id: HM-STD-002
name: Certificate renewal for services outside critical functions
approved_by: AIDR-0952 # change-risk reviewer and class rules
eligible:
service_tiers: [2, 3]
dora_critical_or_important: false
may_touch: [certificate, secret_reference]
limits: {max_resources: 10, deletes: 0, identity_changes: 0, network_changes: 0}
window: "daily 05:00-07:00 Europe/Berlin"
delegation: {draft: "Level 2 Supervised", release: "Level 3 Bounded"}
verification:
probe: external-tls-probe # not run by the pipeline
pass: "chain valid and at least 30 days to expiry"
rollback: {automatic: true, tested_on: 2028-10-04, result: passed}
history: {instances_12m: 41, correlated_incidents: 0}
suspend_if: "two correlated incidents or one in Tier 1 within a quarter"
sampling: {post_release_rereview: 0.10, seeded_errors: 0.05}
expires: 2029-10-04
Appendix A.4 adds the gate as illustrative Rego, which the book states is not executed by its build. The gate denies a change whose declared class is lower than its computed class, a change authored by an agent that has no independent human approver, a release outside the authorised window and a change without a tested rollback. What the record must never contain, in the book's words, is a declared class that overrides the computed class, an agent as its own approver, typed prompts, or a claim of recoverability that the change does not evidence. The case behind the rule is a baseline incident at Hansa Mutual in which a configuration push to the AI gateway routing was treated as a standard change, a canary stage was skipped, and the claims assistant failed in four markets for 47 minutes (INC-HM-2027-0833); the rule that class is decided by the pipeline was written in the problem record that followed.
Practical guide: defining standard change classes
| Field | Content |
|---|---|
| Kind | Method |
| Origin | AI Operations, Chapter 8 |
| Purpose | Defines a class of change that an agent may draft and a pipeline release without per-instance human approval, and keeps it honest over time |
| Use when | A kind of change has been made by hand at least 20 times in twelve months with no change-correlated Tier 1 or Tier 2 incident |
The eight steps are: pick the kind of change; write the class as data; define the verification and test the rollback in staging; set the level per action class, with drafting at Level 1 or 2 and release at Level 3 only when the policy decision point is outside the agent; state the budget consequence; sample after the event; set an expiry of twelve months; and agree the class with the employee representatives where the works agreement requires it. The design choice that matters is the suspension rule. Two change-correlated incidents, or one in a Tier 1 service, within a quarter suspend the class until the change authority has read the post-incident review, so that a class cannot outlive the evidence that earned it. AI Operations, Chapter 8, reports that the DevOps Research and Assessment programme found no evidence in its research that a more formal external review process is linked to a lower change fail rate, and that it recommends peer review supported by automation (10, Nordqvist, 2026); the guide is consistent with that and keeps the independence that Article 17 of Delegated Regulation 2024/1774 requires between those who approve a change and those who request and implement it (235, European Commission, 2024). That is an architectural implication and not legal advice.
Other change records
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Ticket-era and code-era change record comparison | Artefact | AI Operations, Chapter 2 | Two illustrative records (CHG-HM-5870, CHG-HM-6101) showing the shape of each era of operating models; not a template for production, and it uses a fictional service-management product |
| Change record (improvement) | Artefact | AI Purple, Chapter 10 and Appendix A | The change that the improve stage adds for an accepted finding, with owner, due date, cost against the alert budget and a hand-over to verification (CHG-HM-2028-007) |
| Model-change record | Artefact | AI Operations, Chapter 13 and Appendix A.9 | A model change as a normal change with an offline gate, shadow, canary and ramp, graded by a party that does not run the change (CHG-HM-2028-0412) |
| Release record | Artefact | AI Platform and Security, Chapter 8 | Defines a release as the exact set of artefact versions that passed evaluation together, pinned by digest, signed and recorded |
The four records share a prefix or a purpose and are not interchangeable. CHG-* is used by AI Operations for operational change and by AI Purple for a change that closes a finding; the ticket-era pair is fictional and uses a different numbering. The release record of Book 3 answers a different question from either, namely which versions of model, prompt, index, policy and skill were evaluated together, and its manifest lists them with digests, the evidence and a signature; a change that alters any one of them leaves the evaluated combination and needs its own evaluation. The model-change record is the ordinary change record applied to a model: the claims-triage agent's snapshot moves under provider notice, the change is normal and never standard, the change authority does not run the evaluation, and the rollback names both the pinned predecessor and the fallback after its retirement. Its stages, offline gate, shadow, canary assigned by hash of the case identifier, and ramp, are described under progressive delivery below.
Progressive delivery and GitOps
Progressive delivery limits what a bad change can do by controlling who is exposed to it and for how long before it is judged. GitOps keeps the declared state in version control and lets an engine pull it. The two meet at the point where an engine widens a rollout, and the series' conclusion is that the engine must not be the only judge of its own rollout.
Progressive-delivery policy
| Field | Content |
|---|---|
| Kind | Artefact |
| Origin | AI Operations, Chapter 9 and Appendix A.5 |
| Purpose | Defines the rings, bake times, stop conditions and stop path for one service, and names the verifier that is independent of the author, the pipeline and the telemetry path |
| Use when | For every Tier 1 service, before the first staged rollout |
| Inputs | The service record and objective, a ring order, an independent verifier, the freeze calendar |
| Outputs | ROL-* in version control; a seeded-fault test result |
| Identifier family | ROL-HM-* |
| Full template | AI Operations, Appendix A.5 (ROL-HM-001, with an illustrative Rego widening rule and an illustrative SQL change-correlated incident rate) |
The excerpt below keeps the rings, the verifier, two of the five stop conditions, the stop path, the rollback rule and the evidence; the concurrency, window and actor fields, and three stop conditions, are removed.
policy_id: ROL-HM-001
version: 3
service: SVC-HM-001
tier: 1
critical_or_important_function: true
applies_to: [code, configuration, feature_flag] # class never exempts the first ring
rings:
- {id: ring0, target: synthetic-test-tenant, traffic_share: 0.00, bake_minutes: 15}
- {id: cell1, target: market-NO, traffic_share: 0.03, bake_minutes: 60, min_requests: 500}
- {id: cell2, target: market-SE, traffic_share: 0.08, bake_minutes: 120}
- {id: cell3, target: [market-NL, market-PL], traffic_share: 0.34, bake_minutes: 240}
- {id: cell4, target: [market-DE, market-UK, market-CH], traffic_share: 0.55, bake_minutes: 0}
verifier:
observer: OBS-HM-0001
independent_of: [author_identity, pipeline_identity, model_family, telemetry_vendor_path]
probe_vantage: [cloud-b-eu-central, external-synthetic]
max_result_age_seconds: 300
stop_conditions:
- {id: SC1, kind: relative, signal: error_ratio_delta, gt: 0.02, for_minutes: 3, action: rollback}
- {id: SC4, kind: fail_closed, signal: verifier_heartbeat_or_data, missing_minutes: 2, action: hold}
stop_path:
holders: [incident-commander-on-duty, service-owner, agent-operator]
approval_required: false
mechanism: pipeline-freeze-flag-in-store-outside-pipeline-control-plane
target_seconds: 60
rollback:
executed_by: pipeline-controller
rollback_safety_required: true # two-phase for data format changes
irreversible_on_production_data: never_automatic
evidence:
seeded_fault_test: {cadence_days: 90, last_run: 2028-10-18, result: passed, seconds_to_halt: 94}
Four design choices recur through the full record. The first ring is never skipped for any class of change; the verifier is a system other than the releaser, with its own identity, vantage points and store; one stop condition is held out, defined in the verifier's configuration only; and the executor of a rollback is a deterministic controller with a narrow identity, not the agent reversing its own work, so that the rollback is a designed mechanism and not an agent action charged to the autonomy budget. The artefact must never contain a first ring skipped for any class, a verifier that shares identity or model family with the releaser, or an automatic rollback of an irreversible change on production data. The practical guide has nine steps, from ordering the rings to a quarterly report of rollouts started, halted and rolled back, and the date of the last seeded-fault test.
Methods of progressive delivery
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Progressive delivery (rings, canaries, bake time) | Method | AI Operations, Chapters 9 and 13 | Staged exposure with bake times and stop conditions |
| Fail-fast feedback and progressive validation; progressive deployment | Method | Architecture as Code, Chapter 5 | Staged pipeline validation and canary or blue-green rollout |
| Shadow and canary deployment | Method | AI Platform and Security, Chapter 7 | Shadow traffic, canary and champion-challenger rollout for models |
| Feature flags | Method | AI Operations, Chapter 9 | Decouple release from exposure with toggles |
| Rollback safety | Method | AI Operations, Chapter 9 | Check that a rollback is possible before release |
| Model-change release path with pinning and fallbacks | Method | AI Operations, Chapter 13 | Release a model change as a normal change: offline gate, shadow, canary, ramp, pinned versions and fallbacks |
| Practical guide: defining quality objectives and a rollback for an AI service | Method | AI Operations, Chapter 13 | Gives an AI service the objectives, release path and way back that a conventional service has |
The sources for the practices are the operators' own accounts. The Site Reliability Workbook defines a canary as a partial and time-limited deployment of a change and its evaluation, and cautions that the canary must be representative, that metrics must be broken out by population, that the canary is compared with a control and not with the system before the change, and that absolute measures such as SLOs are still needed (227, Warner et al., 2018). Amazon's account of its pipelines reports waves of growing size, a one-box stage at the start of each wave that typically serves at most 10 per cent of the requests of a region or zone, minimum waits between stages, and time windows that exclude nights, weekends and office holidays (229, Liguori, 2020); AI Operations uses it as the parameters of one very large provider, not as a standard. Feature flags are classed by Hodgson as release, experiment, operations and permissioning toggles, with the caution that the artefact must be tested in both states (228, Hodgson, 2017); the series' rule is that a flag flip on a Tier 1 service is a change with a change record, a ring and stop conditions, and that a flag platform in the same failure domain as the service it protects is not a stop path. The AWS guidance on cells describes isolated instances that bound the effect of a failed deployment (231, AWS, 2023), and the account of rollback safety supports the two-phase release of a data-format change, in which every server learns to read the new format before any writes it (230, Pokkunuri, 2019).
Rollback safety is a method because rollback is easy only if the previous version can still read what the new one wrote. The check is written into the change record before release, and a change that drops a column, deletes data or rewrites it in place has no way back; under the Level 4 prohibition of AI Operations, Chapter 4, no agent applies it autonomously. For a model change the method adds three differences: the metric is a graded judgement on a sample, so the canary runs for a minimum number of graded cases and not only a minimum time; assignment is by case, not by request; and shadow comes first. The fallback chain, from the pinned predecessor to a second model to a manual mode, is part of the method, and each link is itself a release with an evaluation result, a date and a stated quality loss. A manual mode that has not been exercised is not recoverable in the series' sense, because the claim needs a test date and a measured time.
GitOps and its engines
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| OpenGitOps principles | Third-party tool (principles of an open project) and method | Architecture as Code, Chapters 7 and 25; AI Operations, Chapters 2, 4, 8, 16, 17, 19 and 32 | Desired state is declarative, versioned and immutable, pulled automatically and continuously reconciled |
| Argo CD | Third-party tool (open-source software) | Architecture as Code, Chapter 7; AI Operations, Chapters 8 to 10, 16 to 19, 32 and Appendices A and B | A pull-based GitOps reconciler with sync policy, sync windows, projects and source integrity verification; protected as Tier 0 |
| Flux | Third-party tool (open-source software) | Architecture as Code, Chapter 7; AI Operations, Chapters 8, 9, 16 to 18, 32 and Appendices A and B | A pull-based reconciler with Kustomization, suspend, tenant lockdown and GitRepository |
| Istio | Third-party tool (open-source service mesh) | Architecture as Code, Chapters 7 and 8 | Named for canary routing and mutual TLS; Istio's own documentation describes a service mesh with percentage-based traffic splitting and a certificate authority that issues certificates for mutual TLS between services (193, Istio (n.d.), n.d.) |
| Argo CD Application manifest | Artefact | Architecture as Code, Chapter 7 | A GitOps reconciliation manifest for a production namespace |
| Istio VirtualService weighted canary routing | Artefact | Architecture as Code, Chapter 8 | A weighted routing manifest for a canary release |
The OpenGitOps principles state the model in four sentences: the desired state is expressed declaratively; it is stored so that immutability and a complete version history are retained; software agents automatically pull it from the source; and software agents continuously observe actual state and attempt to apply the desired state (96, OpenGitOps, n.d.). Kubernetes describes its controllers in the same terms, as control loops that watch the state of the cluster and make or request changes to move the current state closer to the desired state (146, The Kubernetes Authors, n.d.).
The series adds to this a security conclusion that its engines' own documentation supports. A reconciler that applies is an actor in the change system, and its identity is its reach. Argo CD's documentation states that its default role can be narrowed to the namespaces and resources it should manage, and that write access to a trusted repository lets an attacker deploy altered manifests or have resources pruned (164, Argo Project, n.d.); Flux's controllers run with cluster-wide rights by default, and tenants are restricted by service-account impersonation and lockdown flags (166, Flux project, n.d.). AI Operations concludes that the reconciler, its credentials, its repository and the objects that hold its policy are Tier 0 and protected paths, and that an agent reaches the estate by a commit through the change gate, not by a call to a sync interface. A sync policy is the reconciler's mandate, and each setting maps to an action class and a ceiling in the series' own delegation scheme:
| Setting | What the documentation says | Ceiling in AI Operations |
|---|---|---|
| Automated sync | Applies Git when out of sync (162, Argo Project, n.d.) | Level 3 Bounded for a catalogued standard class that passed the gate; otherwise a person triggers the sync |
| Self-heal | Re-syncs when live state deviates; off by default in Argo CD (162, Argo Project, n.d.); Flux corrects at every interval (165, Flux project, n.d.) | Level 3 Bounded for catalogued drift classes outside a freeze; never for the observer's own configuration |
| Prune | Deletes what left Git; off by default in Argo CD, with an allowEmpty guard (162, Argo Project, n.d.) |
A named person for stateful kinds; never Level 4 where it would delete data |
| Sync windows | Cron-scheduled allow and deny windows; deny overrides allow; no matching window means all syncs allowed (163, Argo Project, n.d.) | A second line behind the policy decision point; give each application an allow window |
| Suspend | Flux suspend stops applying revisions and correcting drift (165, Flux project, n.d.) |
Any named holder, no approval; the way back is a Git revert |
Table 16.1: Sync policy as a mandate, after (10, Nordqvist, 2026).
Appendix A.21 of AI Operations gives the Argo CD project that confines a reconciler, the ApplicationSet that generates one Application per cell with the sync policy above (prune: false, selfHeal: true, allowEmpty: false, bounded retry) and, as a design and not a record, a Flux Kustomization for Nordvik Region's portal with prune: false, wait: true and a service account confined to its namespace. Appendix B.21 turns the reasoning into an eleven-point checklist for reviewing a sync policy before it is switched on. These artefacts are not separate entries in the inventory of this chapter, and the two listings of Architecture as Code differ from them in a way that matters. The Argo CD manifest in Architecture as Code, Chapter 7, enables prune: true and selfHeal: true on a production namespace tracking main; it is a minimal illustration of the "as code" framing, not a policy, and under the ceilings above prune would need to be off for stateful kinds. The Istio VirtualService in Chapter 8 splits traffic 90 to 10 between a stable and a canary subset with three retries and a two-second per-try timeout; AI Operations warns that retries are the mechanism of retry storms and advises a retry budget, so the retry values are an illustration to be set deliberately.
Progressive-delivery controllers
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Argo Rollouts | Third-party tool (open-source software) | AI Operations, Chapters 8, 9, 32 and Appendix B | Progressive-delivery controller with analysis, abort and verification, used with an independent verifier's verdict |
| Flagger | Third-party tool (open-source software) | AI Operations, Chapters 8, 9, 32 and Appendix B | Progressive-delivery operator for Kubernetes |
| Kargo | Third-party tool (open-source software) | AI Operations, Chapters 8, 9, 32 and Appendix B | Promotion tool with stages and verification |
Argo Rollouts documents an analysis that makes a rollout continue, abort or pause according to whether a run is successful, failed or inconclusive (167, Argo Project, n.d.); Flagger is described by its project as a progressive-delivery operator for Kubernetes (168, Flagger project, n.d.); Kargo organises promotion through stages, and a failed verification keeps the freight from the next stage (169, Akuity, n.d.) (170, Akuity, n.d.). AI Operations maps the rings, bake times and stop conditions of ROL-HM-001 onto the first and last, and draws the limit. The engines supply the mechanics of hold, abort and promote; they do not supply independence. By default the controller that executes the rollout also starts the analysis, and the template names the metric source and thresholds chosen by whoever writes it. The controller's analysis is therefore a fast first gate that may abort early, and the verdict that permits widening comes from an independent verifier, which the analysis can read through an external metric provider.
Runbooks and remediation
A runbook is the written procedure for a known condition. The tools of this family make it a file that a policy engine, a person and an agent read in the same way, set the ceiling for each class of remediation, and constrain the execution.
Runbook manifest
| Field | Content |
|---|---|
| Kind | Artefact |
| Origin | AI Operations, Chapter 10 and Appendices A.6 and B.7 |
| Purpose | Makes a runbook executable under a bound: preconditions, dry run, action, independent verification, failure path, who may stop it and when it was last exercised |
| Use when | A condition recurs and a procedure is to be run by a machine, or by a person on a machine's schedule |
| Inputs | The alert or objective burn that triggers it, the service record, the delegation profile and mandate, a probe outside the failure domain |
| Outputs | RBK-* in version control, signed on release |
| Identifier family | RBK-HM-*, RBK-NR-* |
| Full template | AI Operations, Chapter 10 and Appendix A.6 (RBK-HM-0007); quoted below |
The manifest below is the worked example of the chapter, a Level 3 Bounded restart of one stateless web instance by the IT-operations agent of Hansa Mutual. It is quoted in full from Appendix A.6.
runbook: RBK-HM-0007
title: Restart one unhealthy stateless web instance
version: 3
service: SVC-HM-005
owner_unit: Group Site Reliability and Incident Management
accountable: Tobias Richter
decision_record: AIDR-0970
action_class: restart
delegation:
profile: DLG-HM-OPS-001
mandate: MND-HM-OPS-001
level: Level 3 Bounded # ceiling for this runbook
never_above: Level 3 Bounded
preconditions:
- target.role == "stateless-web"
- fleet.healthy_after_action >= 4
- fleet.healthy_after_action / fleet.size >= 0.75
- calendar.freeze == false
- runs.last_60_minutes.same_service < 3
- evidence.captured == true # logs and process state saved first
dry_run: required # plan and policy check, no effect
action:
step: restart-instance
targets: 1
max_runs_per_target_per_hour: 1
verification:
probe: synthetic-broker-login
location: probe-eu-b # outside the cluster and the agent's identity
soak_minutes: 10
pass_when: 5 consecutive passes at 60 second intervals
on_failure:
rollback: keep instance cordoned for diagnosis, add one instance (RBK-HM-0009)
then: escalate to incident commander, no second restart
stop:
held_by: [incident commander, agent operator, platform engineer on call]
path: STOP-HM-ITOPS # outside the agent platform, no approval needed
review:
last_dry_run: 2028-10-17
review_due: 2029-01-17
approved_by: Tobias Richter
The preconditions protect capacity and not only correctness: restarting one of five instances is allowed, restarting the last healthy one is not. The probe runs outside the cluster under an identity that the agent does not hold, so the agent cannot make it pass. The failure path ends in escalation and not in a second attempt, because a runbook that retries on its own can keep an unhealthy service in a restart cycle. The artefact must never contain a verification that reads what the action resets, a stop path inside the agent platform, or a ceiling above the delegation profile. A manifest lacking any of the required fields is, in the book's rule, not eligible for a level above Level 1 Assisted. The mandate manifest (MND-*) lists the runbooks that an agent may execute, the runbook states the level, and a policy decision point outside the agent evaluates both and the preconditions at the moment of execution; Chapter 18 treats the mandate.
Practical guide: writing a runbook so that a machine may run it
| Field | Content |
|---|---|
| Kind | Method |
| Origin | AI Operations, Chapter 10 |
| Purpose | Produces a runbook manifest that a policy decision point can enforce, an agent execute and a person audit |
| Use when | Before any runbook is given a level above Level 1 Assisted |
Its ten steps move from naming the condition to recording the level in an AI Decision Record. Three are decisive. The ceiling is set from four questions, reversibility, blast radius, the runbook's record and the window, and the lowest answer governs; a new runbook starts at Level 1 Assisted. The stop is drilled, and the book reports that the first drill of Hansa Mutual's agent took 17 minutes against a target of 5. And the review date is set at 90 days, after which a runbook past its date drops one level.
Remediation tiers and the first ten actions
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Remediation tiers | Method | AI Operations, Chapter 10 | Four tiers from notify to autonomous execution, by who performs and who approves |
| First-ten-actions remediation catalogue | Artefact | AI Operations, Chapter 10 | A table of ten first runbook actions with class, ceiling per organisation and conditions, plus the actions not delegated |
The tiers are notify (Level 0 Advisory), propose (Level 1 Assisted), execute with approval (Level 2 Supervised) and execute within a mandate (Level 3 Bounded); Level 4 is not used for any action of O4 Restore. The catalogue is conservative where an action looks symmetrical but is not. Collecting diagnostics, annotating the incident and notifying the on-call run at Level 3 at Hansa Mutual; restarting one stateless instance, scaling out within quota and renewing a certificate within its renewal window also run at Level 3 under conditions; taking an instance out of rotation, scaling in, rolling back the last deployment and removing regenerable cache files are held at Level 2, because scaling in can remove the capacity the next surge needs and a rollback is safe only while the data model has not moved on. Nordvik Region's column is lower throughout, because it runs no autonomous production agent and its scripted runbooks run at Level 2 with the duty manager's approval. The catalogue also lists what is not delegated at all: silencing, muting or re-thresholding an alert, probe or check; deleting anything not regenerable, and any logs, audit records or backups; changing a runbook, mandate, policy, credential or the stop path; and restricting a named person's access to work. Failover is held at Level 2 at most, restore over existing data at Level 1, and a restart of a stateful primary at Level 2.
The autonomy budget governs the tier. Each Level 3 class carries a tolerance for faulty actions per 28-day window; a failed verification, a rollback or a refusal by the policy decision point counts as one faulty action; a customer-visible effect consumes the whole tolerance; the review is held at 70 per cent consumed; and full consumption lowers the class by one level at once, with raising a named authority's decision. The series' ceiling for a class is therefore a number that moves with evidence, and the catalogue gives its starting value.
Safe execution and signed runbooks
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Safe execution of runbooks (idempotency, retry budget, back-off) | Method | AI Operations, Chapters 10 and 17 | Idempotency keys, retry limits with jitter, rate limits per target, precondition re-reads and compensation steps |
| Signed runbooks and input provenance | Method | AI Operations, Chapter 18 | Sign runbooks and classify inputs by who can write them |
| Restore, do not repair | Method | AI Red Team, Chapters 17, 21 and 22 | Rebuild from known-good definitions and restore into a clean room instead of cleaning in place |
| Drift runbook | Artefact | AI Platform and Security, Chapter 21 | A runbook for input drift of a classifier, linked from an alert |
The five properties of safe execution are preserve before acting, be idempotent, plan before executing, limit the rate and the reach, and roll back the remediation. Each has a source. Google's account of an automation that treated an empty list as everything is the reason an empty, unexpected or oversized input is a refusal (232, Murphy et al., 2017); a caller-supplied request token that lets a service recognise a retry is the usual device for idempotency (233, Featonby, n.d.); Kubernetes documents that a dry-run request passes through admission and validation without persistence, but that authorisation for it is identical to that for the real request, so a dry run checks an action and does not limit the agent (171, The Kubernetes Authors, n.d.); and the advice on cascading failures is randomised exponential back-off, a retry budget per process and no retries at several layers at once (234, Ulrich, 2017). The circuit breaker in the manifest follows: when the same runbook has run against the same service a set number of times in an hour, it stops and calls a person.
Signed runbooks apply the same release discipline to the procedure that an agent executes. AI Operations gives four rules: the runner does not edit the runbook; two people release a change; an unsigned or mismatched runbook is refused and may at most be proposed as advice at Level 0; and retrieval is not authority, so the signed release is what executes. The specification of the Supply-chain Levels for Software Artifacts (SLSA) is cited for the principle that provenance verifies where, when and how something was produced and does nothing unless somebody inspects it (97, SLSA, n.d.). The companion method classifies input text by who can write it into four writer classes, from instruments to external or generated text, and gives a summary the lowest class of what it summarises; an action above notify triggered by text of a lower class needs an independent probe's confirmation or a person's approval.
The drift runbook of Book 3 is the shortest example of an executable procedure, written for an on-call engineer who does not know the classifier well. Its four steps are confirm, assess, contain and close; the contain step raises human review for the affected slice to 100 per cent through a named flag, and the close step records that any threshold change goes through a pull request and never a silenced alert. The runbook is a Markdown file, not a manifest, so it has no preconditions or verification field that a policy engine could read; it shows what a runbook looks like before it is made executable. "Restore, do not repair" is the method that decides where the restored service comes from: a component that an attacker may have touched is rebuilt from code, signed images and versioned configuration by a verified pipeline, because rebuilding requires confidence only in the definition and the pipeline (4, Nordqvist, 2026). Data are restored from immutable backups and are not an agent's decision; that is the last line of AI Operations, Chapter 10.
Runners of runbooks
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Argo Workflows | Third-party tool (open-source software) | AI Operations, Chapters 8, 10, 32 and Appendix B | A container-based workflow engine on Kubernetes for runbook execution |
| Temporal | Third-party tool (open-source software) | AI Operations, Chapters 8, 10, 16, 32 and Appendix B | A durable-execution workflow engine for multi-step runbooks and human-in-the-loop approvals |
Argo Workflows runs each step as a container on Kubernetes, in sequence or as a graph (174, Argo Workflows project, n.d.). Temporal records the action taken for each command of a workflow execution and, after a failure, replays the workflow code against that history (172, Temporal Technologies, n.d.); it also states that an activity is recorded only when it returns, so one that fails to report may run more than once, and recommends idempotent activities (173, Temporal Technologies, n.d.). The engine guarantees that the procedure progresses and not that the effect happened once. AI Operations therefore divides the manifest between the engine and a policy decision point: the engine may enforce what only its operators can edit, such as retry settings and a cancel signal, and everything else, namely level, preconditions at execution time, run counts held outside the engine's own retry, independent verification and revocation of the credential, belongs to a component that the workflow's author cannot reconfigure. The same book extends the point to GitOps engines, which it classes as Tier 0.
Freezes and windows
Change is safest when it happens when somebody can respond to it, and restoring is safest when the rules for acting out of hours are settled in advance.
Freeze calendar and blast-radius budgets
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Freeze calendar | Artefact | AI Operations, Chapter 9 | A calendar of freezes with holders who may authorise an exception, conditions and a record (FRZ-HM-2028 is referred to; no separate template) |
| Freezes and blast-radius budgets | Method | AI Operations, Chapter 9 | Calendar freezes with named exception holders and per-change limits on exposure |
A freeze has a scope, a period, a reason and an owner, and takes one of three forms: a full freeze, a partial freeze that stops high-blast-radius classes while allowing small reversible ones, and a window restriction, which is the permanent form. The calendar is a record in version control read by the policy decision point, so that the freeze is enforced by the pipeline and not remembered by people. The exception rule is to name the holders who may authorise one, require a reason, a named decider other than the author, the smallest ring and the strictest stop conditions, and a review afterwards; the emergency change procedure that the delegated regulation requires, documented, re-evaluated and approved after implementation, is the template (235, European Commission, 2024). For an agent the rule is categorical: no agent at Level 4 releases outside an authorised window or inside a freeze on a critical function, and in practice no agent in either organisation releases inside a freeze at all. A blast-radius budget has four parts, scope, time, concurrency and consumption, and five management measures: escape depth, time to halt, skip rate, change-correlated incident rate and evidence age. Article 5 of the DORA Regulation makes the management body responsible for the arrangements of the ICT risk management framework (197, European Union, 2022), and a body that oversees change risk needs a figure for how far changes get before they are stopped and the date when that was last proved. Chapter 15 treats the alerts and metrics that feed those figures.
Remediation windows
| Tool | Kind | Origin | One-line purpose |
|---|---|---|---|
| Remediation windows by maintenance group and patch-velocity measure | Method | AI Operations, Chapter 11 | A window for vulnerability remediation by maintenance group, with patch velocity measured without gaming |
The structure comes from NIST SP 800-40, which assigns assets to maintenance groups with a plan for routine patching, emergency patching, emergency mitigation and unpatchable assets (98, Souppaya & Scarfone, 2022). The inputs for deciding that a vulnerability is an emergency are the Exploit Prediction Scoring System (80, FIRST, n.d.), the catalogue of known exploited vulnerabilities (81, CISA, n.d.) and the European Vulnerability Database (82, ENISA, 2025). The windows themselves are the design of AI Operations and not a source: the illustrative table for Hansa Mutual sets a routine window of 14 days and an emergency mitigation of 24 hours and patch within 72 hours for exposed Tier 1 assets, longer windows for internal and non-production groups, compensating isolation for legacy platforms, and a ceiling on the agent's deploy action of Level 2 for the two Tier 1 groups, Level 3 for a catalogued routine patch with a tested rollback outside Tier 1 and Level 0 for unpatchable assets. The European instruments leave the number of days to the entity; the delegated regulation requires weekly vulnerability scanning of assets that support critical or important functions and deadlines for patches with escalation when a deadline is missed (235, European Commission, 2024). The measure of patch velocity as a service level is treated in Chapter 15.
Across the degrees of delegation
The tools of this chapter do not change with the usage mode, but the ceiling of each action does. The table is AI Operations, Chapter 8, condensed to the artefacts above.
| Level | O3 Change | O4 Restore | Required outside the agent |
|---|---|---|---|
| Level 0 Advisory | Scores a proposed change for blast radius and freeze conflict | Notifies and points to the runbook | A read-only identity; a score is not an approval |
| Level 1 Assisted | Drafts a change, runbook or policy at a person's request | Proposes the action | Draft only; provenance record |
| Level 2 Supervised | Authors a change and opens a pull request; may apply to non-production | Executes with a named person's approval | A gate on the pull request; a cap on open drafts; seeded errors |
| Level 3 Bounded | Releases a catalogued standard class in a window; reverts catalogued drift | Executes a catalogued, signed runbook within a mandate | A policy decision point outside the agent; an independent probe; a tested rollback; a suspension rule |
| Level 4 Autonomous | Never for an irreversible action on production data, outside a window or inside a freeze on a critical function, for an edit to a protected path, or for any action that disables the observer or the stop path | The same | Not applicable |
Table 16.2: The chapter's tools across the degrees of delegation.
The usage mode describes how the AI is deployed and whose identity it uses; the level describes who decides each action. A Mode 3 agentic work tool drafts infrastructure changes at Level 1 or 2 and never applies them; a Mode 5 autonomous production agent acts at Level 3 only inside a catalogued class or runbook. The tools themselves, the change record, the policy, the manifest and the engines, are the same in every row.
Nordvik Region and Hansa Mutual
Nordvik Region lets an agent draft infrastructure changes and does not let it apply them. Releases happen in windows by push, through the platform team's pipeline, with no continuous reconciler holding production credentials and a read-only drift detector comparing the estate with the declaration. No standard class exists yet that a pipeline releases alone; certificate renewal and scaling inside bounds are the first candidates, and CHG-NR-2028-0007 is a normal change because no class exists. Its scripted runbooks run at Level 2 with the duty manager's approval, and its Flux Kustomization is a design for a possible move to a pull model, not a record.
Hansa Mutual lets standard classes flow to production when the policy checks pass. It runs Argo CD with a sync policy per class and per cell, rings and verifier as in ROL-HM-001, and an IT-operations agent that acts at Level 3 Bounded inside catalogued runbooks such as RBK-HM-0007 and at Level 2 for configuration change, so that a change of the kind that broke the AI gateway routing in INC-HM-2027-0833 is authored at Level 2 and reviewed by a named person. Its change-risk reviewer scores proposed changes at Level 0 Advisory and its score informs the human decision without replacing it.
Treatment of the inventory
The inventory assigns 52 entries to this chapter. Forty are treated: five in entries of their own (the change class and record, the standard-change guide, the progressive-delivery policy, the runbook manifest and the runbook guide) and thirty-five in the family tables and the prose of the families, where the Book 1 and Book 10 entries for Argo CD, Flux and the OpenGitOps principles are one row each. Twelve are reassigned: the CALM definition example (Chapter 4); the Kubernetes, Helm, Compose and Terraform cluster examples (Chapter 6); the unified-build-release.yml workflow excerpt, the deployment policy in Rego and the pipeline-setup and reproducible-release guides (Chapter 5); the cloud versus on-premises decision matrix (Chapter 6 or 9); the deployer obligations checklist (Chapter 8); the readiness assessment framework (Chapter 3); decoys and pre-release sycophancy testing (Chapter 19); the reading of a system card (Chapter 23); and the second-hypothesis rule (Chapter 12). Appendix A holds every entry. Appendix A.21 of AI Operations is an artefact of this chapter that the inventory did not assign; it is recorded in the findings.
Summary
This chapter collected the tools that let an operator change a service and bring it back. They have one structure in common: the action is bounded by a record that is data, evaluated by something other than the actor, and stoppable from outside. The change class and record bound a change by the computed class of the diff; the progressive-delivery policy bounds its exposure by rings and bake times; the runbook manifest bounds a remediation by preconditions, a dry run and a verification; the sync policy bounds an engine; and the freeze calendar and windows bound the time.
Three conclusions follow. First, independence is the recurring requirement and no engine supplies it. A progressive-delivery controller can abort a rollout but should not be the only judge of it; a workflow engine can resume a procedure but cannot guarantee that an effect happened once; a GitOps engine reconciles what the declaration contains and nothing else. The verifier, the policy decision point and the stop path are separate components, and the tools that name them are the tools that matter most. Second, a ceiling is a number that moves with evidence: a class is suspended on two correlated incidents, a runbook drops a level when it is past its review date, and a Level 3 class falls a level when its autonomy budget is consumed. A tool that states its ceiling without a suspension rule has stated a hope. Third, the series records differences that the reader should know. The identifier CHG-* serves operational change, improvement of a detection and a fictional ticket-era record; the Argo CD manifest of Architecture as Code enables prune on a production namespace, which the later ceilings would not permit; and one artefact of this chapter, the reconciler sync policy, is in an appendix that the inventory did not assign. Chapter 25 draws these together. Chapter 17 continues with the tools for capacity, cost and energy, which bound the resources that these actions consume.
Sources
- Nordqvist, G. AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (4, Nordqvist, 2026)
- Nordqvist, G. AI Operations: How Artificial Intelligence Transforms IT Operations in European Enterprises and Public Authorities. (10, Nordqvist, 2026)
- Forum of Incident Response and Security Teams. Exploit Prediction Scoring System (EPSS). (80, FIRST, n.d.)
- Cybersecurity and Infrastructure Security Agency. Known Exploited Vulnerabilities Catalog. (81, CISA, n.d.)
- European Union Agency for Cybersecurity. European Vulnerability Database (EUVD). (82, ENISA, 2025)
- OpenGitOps. GitOps Principles, version 1.0.0. (96, OpenGitOps, n.d.)
- SLSA. SLSA specification, version 1.2 (Build: Provenance and Build: Verifying artifacts). (97, SLSA, n.d.)
- Souppaya, M. Guide to Enterprise Patch Management Planning: Preventive Maintenance for Technology. (98, Souppaya & Scarfone, 2022)
- The Kubernetes Authors. Controllers. (146, The Kubernetes Authors, n.d.)
- Argo Project. Automated Sync Policy. (162, Argo Project, n.d.)
- Argo Project. Sync Windows. (163, Argo Project, n.d.)
- Argo Project. Security. (164, Argo Project, n.d.)
- Flux project. Kustomization. (165, Flux project, n.d.)
- Flux project. Flux multi-tenancy lockdown. (166, Flux project, n.d.)
- Argo Project. Argo Rollouts: Analysis & Progressive Delivery. (167, Argo Project, n.d.)
- Flagger project. Flagger: Progressive Delivery Operator for Kubernetes. (168, Flagger project, n.d.)
- Akuity. Kargo's Core Concepts. (169, Akuity, n.d.)
- Akuity. Verification. (170, Akuity, n.d.)
- The Kubernetes Authors. Kubernetes API Concepts. (171, The Kubernetes Authors, n.d.)
- Temporal Technologies. Temporal Workflow Execution overview. (172, Temporal Technologies, n.d.)
- Temporal Technologies. Activity Definition. (173, Temporal Technologies, n.d.)
- Argo Workflows project. Argo Workflows: the workflow engine for Kubernetes. (174, Argo Workflows project, n.d.)
- Istio. Architecture. (193, Istio (n.d.), n.d.)
- European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (197, European Union, 2022)
- Warner, A. Canarying Releases. (227, Warner et al., 2018)
- Hodgson, P. Feature Toggles (aka Feature Flags). (228, Hodgson, 2017)
- Liguori, C. Automating safe, hands-off deployments. (229, Liguori, 2020)
- Pokkunuri, S. Ensuring rollback safety during deployments. (230, Pokkunuri, 2019)
- Amazon Web Services. Reducing the Scope of Impact with Cell-Based Architecture. (231, AWS, 2023)
- Murphy, N. The Evolution of Automation at Google. (232, Murphy et al., 2017)
- Featonby, M. Making retries safe with idempotent APIs. (233, Featonby, n.d.)
- Ulrich, M. Addressing Cascading Failures. (234, Ulrich, 2017)
- European Commission. Commission Delegated Regulation (EU) 2024/1774 of 13 March 2024 supplementing Regulation (EU) 2022/2554 with regard to regulatory technical standards specifying ICT risk management tools, methods, processes, and policies and the simplified ICT risk management framework. (235, European Commission, 2024)