The position, stated up front
Everything below is organized around that claim.
Self healing systems AI: the loop, in five parts
Most products stop at three. The last two are what make the first three safe to run without asking.
Why diagnosis is the easy half
Diagnosis is a retrieval and correlation problem, and both are well understood. Pull the last deploys, the dependency status, the error rate by endpoint, the resource trend, and the previous incidents with a similar signature. Rank the candidates. Most production faults resemble something that has happened before, which is why a good runbook library outperforms a clever model.
That also names its own decay. The diagnosis step is retrieval over runbooks and past incidents, so its quality degrades exactly as fast as that corpus rots. Runbooks describing a service you retired last spring are actively harmful: they retrieve well, read confidently and point at the wrong thing. Keeping the corpus current is an ongoing job, and it is the same class of problem covered in RAG or fine-tuning.
Even so, a wrong diagnosis is survivable. A wrong action on production is not, and that asymmetry should decide where your engineering effort goes.
Verification is a predicate, written first
Here is the rule that changes the safety profile of the whole system: the success condition is written before the action runs, and it is machine-checkable.
Not “the service looks healthy”. A predicate. Error rate for this endpoint back below its pre-incident baseline, sustained for a defined window, with request volume above a floor so the check cannot be satisfied by silence. That last clause is the one people forget, and it is how a system declares itself recovered while serving nothing.
Verification also needs a clock. If the predicate is not satisfied inside a stated window, the action is treated as failed, whatever the logs say. Waiting indefinitely for a fix to work is the automated version of hoping.
Rollback, and the concept of a known state
Every automated action carries its own reversal, and the reversal is tested. If an action cannot be reversed, it does not go in the automated set. It goes behind a human approval, permanently and by design.
The purpose of rollback is not just to undo harm. It is to guarantee that whatever a responder eventually picks up is in a describable state. The worst incident to inherit is one where something has been done, twice, partially, and nobody can say what the current configuration is. An agent that tried three fixes and reversed each one leaves a clean timeline. An agent that tried three fixes and left them all in place has produced a new and original system nobody has ever operated.
Named failure modes
Six, each with the guardrail that answers it.
Notice that not one of those guardrails is about model quality. They are operational engineering, and they are the reason we treat this as infrastructure work rather than an AI product.
What it must never do without a person
The boundary we write into every build, before anything is connected:
The approved action library is the single most important artifact here. It is a written list, agreed with your operations team, of the actions the agent may take and the preconditions for each. It grows slowly, by promotion: an action that a human has run repeatedly, that has a clean verification predicate and a tested reversal, becomes a candidate. Nothing arrives in the library because a model suggested it.
- No action outside the approved action library. A model never composes an arbitrary command against production.
- No irreversible action. Nothing that deletes data, drops a table, revokes access permanently or touches a backup.
- No action during a declared change freeze, without an explicit override by a named person.
- No action on a system flagged as regulated or customer-facing beyond a stated tier, unless a human approves it in the path.
- No silent action. Every attempt is logged with its evidence, its predicate, its result and its reversal status, whether it succeeded or not.
A night in the life
An alert fires at 03:14 on rising error rates. The Self-Healing Infra Agent corroborates against a second signal, captures diagnostics, and correlates a deploy that landed 40 minutes earlier with the endpoint now failing.
The candidate action is a rollback of that release. It is in the library, it is reversible, and the affected service is inside the allowed tier. The agent writes its predicate: error rate for the endpoint back under baseline for five sustained minutes at normal request volume. It rolls back one region, waits, and checks.
The predicate holds, so it widens to the remaining regions one at a time, verifying each. Then it files the incident record with the full timeline, links the offending change, and pages nobody, because there is no longer anything for a person to do at 3 a.m. The engineer reads it at 08:30 with the evidence assembled.
Had the predicate failed, the rollback would have been reversed, the system returned to its pre-action state, and a human paged with everything gathered. That is the same outcome you get today, minus the twenty minutes of gathering.
How the action library grows
Promotion, never suggestion. An action becomes a candidate when three things are true: humans have run it repeatedly for the same signature, it has a verification predicate somebody has written down, and its reversal has been tested on a system that matters.
The candidate then runs in shadow for a period. It proposes, a human decides, and the gap between the two is recorded. When the proposals match what the responders actually did, at a rate agreed in advance, the action moves into the automated set for that signature and that service tier alone. Not for every signature that looks similar.
This is slow on purpose. A library of six actions that are always correct is worth more than forty that are usually correct, because the value of automated remediation is entirely in the trust it earns. One bad automated action at 3 a.m. costs more confidence than fifty good ones build.
The honest limits
Built with those limits stated, this is a durable piece of infrastructure. That is what we build as self-healing infrastructure agents.
The short version
- Diagnosis is the easy half. Verification and rollback are what make automated action safe.
- The success predicate is written before the action, is machine-checkable, and includes a volume floor.
- Any action without a tested reversal stays behind human approval, permanently.
- Restrict the agent to an approved action library that grows by promotion, never by suggestion.
- Guard against symptom suppression, evidence destruction, flapping, blast radius and bad signals.
- A system that heals the same fault nightly is reporting a defect, not solving one.
Keep reading
- RAG or fine-tuning? A decision guide that isn’t vendor marketing - Why runbook retrieval decays, and what to do about it.
- Your AI pilot worked. Here’s why production won’t. - The gaps between a working demonstration and a system on call.
- Questions, answered - Cost, safety, timelines and what stays human.
