This guide is for the CTO or engineering lead who is the first call when a live agent does something wrong, and for the operations or risk owner who decides whether it returns to production. Our AI automation guide decides which process gets an agent, and the agentic AI automation service covers delivery; this page starts once an agent has already acted on a live system in a way nobody intended.
In this guide
- What should happen in the first 15 minutes after an AI agent acts wrong?
- Not every action can be undone: test reversibility before you promise a fix
- The containment-to-recovery runbook
- Five outcomes, and the conditions for each
- Where AI fits
- What delivery record exists, and what does not
- Alternatives: who runs incident response for an agent
- Limits of this guide
- Frequently asked questions
- Next step
What should happen in the first 15 minutes after an AI agent acts wrong?
The first 15 minutes are for containment, not diagnosis: stop the agent, revoke its credentials and tokens, and confirm it cannot repeat the action while anyone investigates. A control that only silences the agent's logging has not contained anything; the stop has to sit in the agent's access, not just its visibility, because a model cannot be trusted to comply with a shutdown instruction inside its own prompt. Before anything is reversed, preserve the run: prompt and model version, every tool call and the records it touched, the permissions it held, and the final output. An agent incident is defined by its business consequence, not by whether the system technically "failed" — a completed task that reached a customer or a ledger with the wrong result is already an incident.
Our observability and kill-switch guide designs the trace and the stop this phase depends on; this page starts at the moment that stop has already been pulled, or should have been. Our operating model guide names who is paged when containment is needed — the operations owner's job is this phase, the risk and compliance owner's job starts in the next one.
Not every action can be undone: test reversibility before you promise a fix
Rollback feasibility depends entirely on what the agent did. Internal, controlled-system writes are often reversible outright; actions that reached outside the system — a sent email, a submitted payment, a message a customer already read — can usually only be compensated for, never undone; some outcomes, such as exposed restricted data, are neither. Sort every affected action into one of these three classes before promising anyone a "rollback."
| Class | What it covers | What recovery looks like | Example |
|---|---|---|---|
| Technically reversible | A write inside a system the agent and your team fully control | Restore the prior value directly, using the action's own identifier so a retry does not undo someone else's later, legitimate change | An agent updates a customer record's status field; the prior value is restored from the stored before/after pair |
| Compensable only | An action that left the system of record and reached a person, a ledger or a third party | A new, opposite action plus, often, a disclosure — not a restore | An agent sends a duplicate invoice; the fix is a cancellation notice and a correction email, not deleting the sent message |
| Neither reversible nor compensable | Data already read by someone outside the company, or damage to trust and reputation | Containment, disclosure and a documented response; no technical action undoes it | An agent exposes one customer's record to another; the record cannot be "unseen," so the response is procedural, not technical |
Decide which class an action falls into before deciding how urgently to act: a technically reversible error caught in minutes is a very different incident from a compensable one that has already reached a customer, even if both started from the same bug.
The containment-to-recovery runbook
-
Contain
Stop the agent, revoke its credentials and tokens, and confirm — not assume — that it cannot act again. A kill switch that only stops new log entries has not contained anything.
-
Preserve evidence
Capture the full run before changing anything: prompt and model version, every tool call and its result, the permissions in force, and every record touched. This is what turns "something went wrong" into an answerable question of which step, tool and retry caused it.
-
Assess reversibility
Sort every affected action into the three classes above, and map the blast radius: what changed, what left the system, and whether downstream automation fired before the error was caught.
-
Act on each class separately
Restore what is technically reversible using the action's own identifier, so a retry cannot overwrite a later, legitimate change; issue the compensating action and any required disclosure for what is only compensable; for what is neither, move straight to containment and disclosure.
-
Find the failed control layer, not just the wrong output
A corrected output is not a corrected system. Ask which layer let the action happen: workflow design, permission boundary, tool behaviour, input data, prompt or model, monitoring, or the approval step that should have caught it. Our human-in-the-loop controls guide designs that checkpoint layer; an incident here usually means one was missing or misplaced.
-
Decide the agent's fate
Choose one of five outcomes, below, before the agent runs again against live data.
-
Turn the incident into a test case
Add the failure scenario to the evaluation suite that gated the agent before launch, so the next release catches it before a repeat.
Five outcomes, and the conditions for each
An agent that has caused an incident does not default back to "running normally." Choose deliberately among five outcomes, each with its own re-enablement bar:
- Return it when the cause is identified, fully corrected, and the issue was low-risk: verify the fix in shadow or approval mode first, not straight back to full autonomy.
- Restrict it when the agent earns its place but had too much authority: narrow its permission scope or add an approval gate for the action class that failed.
- Roll back when a recent change — a prompt edit, a model update, a new tool or permission — caused the problem: revert to the last known-good, version-pinned configuration.
- Redesign it when the workflow, the data it relies on, or the tool structure is weak: a point fix will not hold, and the task needs breaking down differently.
- Retire it when the incident shows the agent produces more risk than value for this process: ending one is a legitimate outcome, not a failure to fix it.
Before returning a contained agent to any of the first three outcomes, confirm: the cause is documented, credentials touched by the incident are rotated, the audit trail is complete end to end, the rollback path has been tested rather than assumed, the fix passes in shadow or approval mode under real traffic, and a named person owns post-return monitoring. Skipping this checklist to resume faster trades a verified fix for an unverified repeat of the same failure.
Where AI fits
Netbase works with the major commercial and open-source AI tools and models, chosen per project, and treats a tested rollback path and a documented incident runbook as part of what makes an agent fit to operate, the same way the observability guide treats a kill switch — a control that has to exist and be rehearsed before the first real incident, not assembled during one.
-
In delivered work
AI content moderation on a classifieds marketplace
A testing and optimisation phase ran before the flagging model reached the admin review dashboard, for a client that is not named; the same staged-verification discipline is what a safe return to production needs after an incident.
-
Available capability
Incident runbook and rollback-path design
Building the reversibility test, the five-outcome decision point and the re-enablement checklist into an agent's operating model, as part of agentic AI automation delivery; not yet tied to a published case that names this instrumentation specifically.
What delivery record exists, and what does not
What exists. Netbase has delivered AI projects for clients that are not named, where milestones include a testing and optimisation phase before a system reached its review dashboard or went live: an AI content moderation model for a classifieds marketplace (F113) and a WhatsApp AI chatbot with CRM integration (F115), alongside anonymised RAG, document AI and MLOps engagements (F072). Netbase's own security practice includes role-based access control, MFA and vulnerability scanning (F021), the controls an incident's containment and credential-revocation steps rely on, and Netbase holds ISO 27001 certification and a SOC 2 Type II attestation for its own operations (F022, F051, F052).
What does not exist. No published Netbase record names an incident count, a resume time, a rollback count, or a root-cause time-to-detect for any agent. No record describes a live agent that has gone through this exact contain-preserve-assess-decide sequence in production. Netbase's certifications cover Netbase's own operations; they are not extended to a client's agent, its data or its rollback path.
Alternatives: who runs incident response for an agent
| Approach | Strength | Weakness | Choose it when |
|---|---|---|---|
| The engineering team that built the agent handles incidents ad hoc | Fast to start, no new process | No tested rollback path or fixed decision point; relies on memory under pressure | A single, low-risk, internal-only pilot agent |
| A written runbook owned by the operations role named in the agent's operating model | Repeatable, rehearsed before the first real incident | Needs upfront work to write the reversibility table and test the rollback path | Any agent that writes to a live system or is customer-facing |
| A delivery partner builds and maintains the runbook and recovery path | One team across the agent, its guardrails and its recovery path | Needs the same ownership questions as any delivery engagement | No internal owner exists, or several agents need one consistent pattern |
In Netbase custom development the client owns the IP created for it, and most projects run on fixed-price contracts agreed after discovery, with milestone-based, monthly team retainer and KPI-linked terms also offered where they fit better (F067, F069).
Limits of this guide
- This is an engineering and operations runbook, not legal, regulatory or compliance advice; what a specific incident requires by law depends on the data, sector and jurisdiction involved.
- No figure in this guide is a benchmark: no incident count, resume time or rollback success rate is published, because none is registered as a Netbase fact.
- The reversibility classes and the five-outcome decision are a starting structure, not a certified framework; a company with regulatory reporting obligations should confirm its own disclosure requirements separately.
Plan the next step with a Netbase consultant
Frequently asked questions
No. An internal database write is often reversible directly, but an action that left your systems — a sent message, a processed payment, a notification a customer already saw — can usually only be compensated for, and data another person has already seen cannot be undone by any technical step.
No. A fix should pass in shadow or approval mode under real traffic first, with credentials rotated and the rollback path actually tested; resuming immediately trades a verified fix for an unverified repeat of the same failure.
Restricting narrows what an otherwise sound agent may do going forward, such as adding an approval gate for the action class that failed; rolling back reverts the agent's own configuration — prompt, model version or tool permissions — to the last version known to work.
Next step
Tell us which agent is live, or about to be, and the systems it can write to, and we will book a solution review to design its reversibility classification, its recovery runbook and its re-enablement checklist. You can also see the enterprise AI agent platform solution or browse more Netbase insights.
Related services and solutions
AI automation and agents that keep people in charge
Netbase provides AI automation and agent development for operations teams that want repetitive, multi-step work done by software while people keep approval over the decisions that matter. We combine rule-based automation with AI where it adds value, build in human approval, and measure return against a baseline.
Learn More
Enterprise AI agent platform: run AI agents with approvals, audit logs and cost limits
An enterprise AI agent platform is a governed runtime where AI agents plan and carry out multi-step work through approved tools, while approvals, audit logs, access rules and cost limits keep every action accountable. Netbase offers it as a forward-looking solution, built inside your own environment rather than licensed as a product.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.