REFERENCE SCENARIO · SAAS & TECHNOLOGY
Reference scenario - composite, not a named client. A composite build illustrating our method. Figures are modeled and the model is shown.
This self-healing devops case study is a reference scenario. No engineering team sits behind it. It shows how the runbook-shaped share of a night’s alerts can be absorbed by agents while the new ones still page a person, and how each figure here is calculated.
What this page is not
There is no client behind this page. Nobody has given us permission to publish their incident data, and incident data is among the last things a company wants published.
So this is a composite, declared as one. The alert classes are real categories. The volumes, the class mix and the minutes per page are inputs chosen to make the arithmetic visible, and not one is a measurement.
The situation this build answers
Alerts reach a person because a threshold was crossed, not because a decision is needed. Most of the time the response is already written down and the engineer is performing it, not choosing it.
At three in the morning that distinction matters more than at three in the afternoon. The cost of a page is not the fix. It is the sleep and the hour after it.
Where the time actually goes
- The same handful of alert classes recur, and each is answered from scratch by whoever is awake.
- Runbooks exist, and following one is manual labor performed by a tired person.
- Diagnosis starts with gathering: logs, recent deploys, resource state, the last similar incident.
- A quietly recurring fault is never counted, so it is never fixed.
- Out-of-hours pages cost the following day as well as the night.
The hive, agent by agent
Self-Healing Infra Agent
Takes the alert first, gathers the evidence, matches it to a known class and applies the approved remediation. Then it verifies the symptom actually cleared.
Incident Response Agent
Owns anything that becomes an incident: the timeline, the comms draft, the affected surface, and the handover to whoever is paged.
The orchestrator
Decides whether an alert is a repair, an incident or a page, and enforces the attempt ceiling that stops a loop.
Nothing here is a silent fixer. Every remediation writes a record, and the recurrence count for each class is reported weekly. The services behind this build are self-healing infrastructure AI agents and, because a system like this is operated rather than delivered, managed AI agents.
What the agents may not do
Autonomy boundary
Agents may restart a service, clear a cache, scale within a stated ceiling, rotate a node and roll back a release whose gates passed. They may not run a migration, touch a secret, change a scaling policy or edit infrastructure code.
Attempt ceiling
Two failed remediations stop the agent and page a person. There is no third try.
Reversibility
Every automated action has a defined reversal, and an action without one is not in the playbook.
What stays human
Novel failures, anything touching data, customer comms during a live incident, and the decision that an incident is over.
Logging
Every observation, inference, action and verification is timestamped, which doubles as the post-incident timeline. Read our governance approach for how that is held.
How the build would run
| Phase | Typical | What happens |
|---|---|---|
| Audit | 3-10 days | Ninety days of alerts are classified to find the real runbook-shaped share and which classes recur. |
| Diagnose only | 2 weeks | The agent gathers evidence and proposes a remediation on every alert. It applies none. |
| Repair, class by class | 4-8 weeks | Classes move to automated repair one at a time, each with its own kill switch and attempt ceiling. |
| Managed | ongoing | Playbooks are maintained, the recurrence report is worked, and new classes are added on evidence. |
How to read this self-healing devops case study
Every figure below is arithmetic on stated inputs. The multiplication is shown so the argument happens at the assumption, not at the conclusion.
| Input | Assumed value |
|---|---|
| Alerts reaching on-call | 640 per month |
| Out-of-hours share | 35% |
| Runbook-shaped share | 62% |
| Needs a human decision | 23% |
| Genuinely new | 15% |
| Engineer time per in-hours page | 35 minutes |
| Engineer time per out-of-hours page | 55 minutes |
| Daily read of the remediation log | 10 minutes, 22 working days |
The modeled outcome, with the arithmetic
| Figure | The arithmetic | Label |
|---|---|---|
397 runbook-shaped alerts | 640 × 62% | Modeled |
243 still paging a person | 640 − 397 (147 decisions + 96 new) | Modeled |
224 out-of-hours pages today | 640 × 35% | Modeled |
85 out-of-hours pages after | 224 × 38% | Modeled |
448 h engineer time today | (416 × 35 min) + (224 × 55 min) = 26,880 min | Modeled |
~174 h engineer time after | (158 × 35 min) + (85 × 55 min) + (22 × 10 min) = 10,425 min | Modeled |
~274 h returned per month | 448 − 173.8 | Modeled |
~61% less engineer time | 274.25 ÷ 448 | Modeled |
100% of actions logged and reversible | Design rule, not a rate | Target |
2 remediation attempts, then a person | Design rule, not a rate | Target |
What we would measure, and what we will not claim
We publish no outcome range for this build. No consented client result exists behind it, and we will not invent one to fill a template slot.
What we would change next time
Treat the recurrence report as the deliverable. An agent that repairs the same fault forty times is hiding a defect that a person would have escalated. We would call this build a success even if page volume held, provided those forty repairs became visible.
Fix the thresholds before automating the response. A meaningful share of any alert backlog is not a fault at all. Automating a reply to a bad alert makes the bad alert permanent.
The parts this build is made of
Self-healing infrastructure AI agents
Detection, diagnosis, approved repair and escalation.
Managed AI agents
Because playbooks, evals and ceilings need operating, not handing over.
Self-Healing Infra Agent
The repair loop and its ceiling.
Incident Response Agent
Timeline, comms draft and handover.
AI agents for SaaS and technology
Where else agents pay off in this sector.
