Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

AI Agent Change Management and Versioning: How to Update a Live Agent Without Breaking What Already Works

Every change to a live AI agent — a new prompt, a model version, an updated tool or a wider permission — is a release: give it a version identifier, gate it with the same regression set that cleared the agent for launch, roll it out in stages, and keep the prior version ready to restore if it regresses.

Book a solution review See the related solution

By Huy Nguyen (David), Founder & CEO · Updated · 11 min read

star

This guide is for the CTO or engineering lead who has to approve an agent's next prompt, model or tool change, and for the operations owner who watches a staged rollout and decides whether to widen it. Our AI automation guide decides which process gets an agent, and the agentic AI automation service covers delivery; this page is the standing release process that keeps a live agent safe to change after it is already working.

In this guide

What counts as a change to a live AI agent, and does it need a release process?

Five things change an agent's behaviour, and each needs its own version identifier: the prompt or system instructions, the model version it calls, a tool or MCP server configuration, the permission scope it holds, and the workflow structure around it (a new step, a new branch in the logic). A sixth kind of change is easy to miss because nobody on the team made it: a model provider updates the underlying model behind a fixed API endpoint, which changes the agent's behaviour exactly as a deliberate model-version bump would, and needs the same gate even though no pull request was opened. Treat "nothing changed in our repo" as a false signal; check the model version actually in use before ruling out a release as the cause of a regression. Our drift detection guide covers how to notice that sixth kind of change before a user does.

A change with no version identifier cannot be compared against what came before it, cannot be rolled back cleanly, and turns an incident investigation into guesswork about which of several recent edits caused it. Versioning is not bureaucracy layered onto a working agent; it is what makes "what changed" an answerable question the next time something looks different, the same discipline our incident response and rollback guide assumes already exists when it says to "revert to the last known-good, version-pinned configuration."

Match the gate to the change: a comparison

Not every change needs the same ceremony. A wording-only prompt edit that does not touch scope or tools can clear a lighter gate than a model version bump or a new permission; match the two, not the calendar, to how much the change can alter behaviour.

Change class Example Minimum gate before any live traffic Rollout path
Wording-only prompt edit Clarifying an instruction, fixing a typo, rephrasing a response template Full run against the existing regression set (no new cases required) Canary (small traffic share), then full promotion within hours if stable
Prompt edit that changes scope or behaviour Adding a new instruction, changing what the agent is allowed to say or do Regression set plus new test cases for the changed behaviour Shadow mode first, then canary, then full promotion
Model version change (including a silent provider-side update) Upgrading to a new model release, or discovering the provider already moved one Full regression and adversarial set rerun, compared against the frozen baseline Shadow mode required; canary with a named reviewer before promotion
Tool or MCP configuration change A new tool, a changed API contract, a different retrieval source Regression set plus tool-specific test cases; verify the tool's own access scope separately Shadow mode, then canary, then promotion with the tool owner's sign-off
Permission scope change Widening what the agent may write to, or which systems it can reach Regression set plus every test case tied to the newly reachable action Canary under close monitoring; never a same-day full promotion

A change that fails its gate does not reach any live traffic; it returns to the draft stage, the same separation our evaluation harness guide uses for a brand-new agent, now reused for every change to one already live.

The release ladder for a live agent

  1. Version it before you touch it

    Give the prompt, model reference, tool configuration or permission set its own identifier the moment a change is proposed, stored separately from the version currently live. An editable draft never overwrites the running configuration.

  2. Run it in shadow

    Send the new version a copy of real traffic without letting its output reach anyone, and compare its behaviour against the current live version on the same inputs. This is where a silent model-provider update first becomes visible, before it reaches a single real user.

  3. Gate it against the regression and adversarial set

    The new version has to clear the same evaluation suite that gated the agent's original launch, plus any case added from a prior incident. A version that fails stays in shadow regardless of how promising it looked informally.

  4. Promote it to a canary share of traffic

    Route a small, defined percentage to the new version, with the operations owner watching the traces and cost signals our observability guide designs, before widening further.

  5. Widen or stop, deliberately

    Promotion to full traffic is a decision a named person makes from canary evidence, not a default after a fixed number of hours; a canary that shows no regression still needs that explicit sign-off.

  6. Keep the prior version pinned, not deleted

    The version a new release replaces stays available and restorable for a defined window, so a rollback is a configuration switch, not a rebuild from memory.

  7. Log the release

    Record what changed, who approved it, the regression results it cleared and the date it reached full traffic — the version history that both the evaluation harness and an incident runbook need to already exist when either is used.

Where AI fits

Netbase works with the major commercial and open-source AI tools and models, chosen per project, and treats a versioned release process — not a direct edit to a live agent's configuration — as part of what keeps an agent trustworthy to change, the same discipline our observability guide applies to watching a running agent and our incident response guide applies to recovering from one.

What delivery record exists, and what does not

What exists. Netbase has delivered AI projects for clients that are not named, structured in milestones that include a dedicated testing and optimisation phase before a build reached its review dashboard or went live: an AI content moderation model for a classifieds marketplace (F113) and a WhatsApp AI chatbot with CRM integration delivered through design, integration, NLP tuning, testing and optimisation (F115), alongside anonymised RAG, document AI and MLOps engagements (F072). Netbase's own practice includes secure code review and version control and role-based access control (F021), the baseline a release's version history and access boundaries rely on, and Netbase holds ISO 27001 certification and a SOC 2 Type II attestation for its own operations (F022, F051, F052). Netbase works with models from multiple commercial and open-source providers, chosen per project (F070, F073).

What does not exist. No published Netbase record names a rollout percentage, a release count, a regression pass rate or a time-to-promotion for any agent. No record describes a live agent that has gone through this exact version-shadow-canary-promote sequence in production. Netbase's certifications cover Netbase's own operations; they are not extended to a client's agent, its versioning system or its release pipeline.

Alternatives: who runs the release process for an agent

Approach Strength Weakness Choose it when
Whoever built the agent edits it directly when a change is needed Fast to start, no new process No version history, no shadow or canary stage; a bad change reaches every user at once A single, low-risk, internal-only pilot agent with no live users yet
A written release process owned by the technical owner named in the agent's operating model Repeatable, reuses the same regression set the agent launched with Needs upfront work to define the change taxonomy and build shadow/canary routing Any agent that writes to a live system or is customer-facing
A platform's built-in draft/published versioning feature Version history and environment tags come with the tool Versioning is scoped to that platform; moving agents or providers later needs its own exit plan The agent already runs on a platform with this feature, and the vendor lock-in and exit guide terms are acceptable
A delivery partner builds and maintains the release process One team across the agent, its evaluation set and its release pipeline Needs the same ownership and sign-off questions as any delivery engagement No internal owner exists, or several agents need one consistent release discipline

In Netbase custom development the client owns the IP created for it, and most projects run on fixed-price contracts agreed after discovery, with milestone-based, monthly team retainer and KPI-linked terms also offered where they fit better (F067, F069).

Limits of this guide

  • This is an engineering and operations practice, not legal, regulatory or compliance advice; what a specific change requires to document or disclose depends on the data, sector and jurisdiction involved.
  • No figure in this guide is a benchmark: no rollout percentage, promotion time or regression pass rate is published, because none is registered as a Netbase fact.
  • The change taxonomy and release ladder are a starting structure, not a certified framework; a provider's own release notes and API behaviour should be checked directly, since model updates are not always announced on a fixed schedule.

Plan the next step with a Netbase consultant

Frequently asked questions

No. A wording-only edit that does not change scope or behaviour can run the existing regression set and move to canary quickly; a change that alters what the agent is allowed to say or do needs shadow mode first, and a model or permission change needs the full ladder.

Yes. If the model behind a fixed API endpoint changes, the agent's behaviour can change even though nobody edited a prompt or wrote code; check which model version is actually in use and run it through the same regression set as any other model-version change.

Shadow mode runs the new version alongside the live one without letting its output reach any real user, purely to compare behaviour; a canary rollout sends a small, defined share of real traffic to the new version and measures it live, before deciding whether to widen it.

Next step

Tell us which agent is live, and the prompt, model or tool change you are planning next, and we will book a solution review to design its version taxonomy, its regression gate and its staged rollout. You can also see the enterprise AI agent platform solution or browse more Netbase insights.

AI automation and agents that keep people in charge AI automation and agents that keep people in charge

Netbase provides AI automation and agent development for operations teams that want repetitive, multi-step work done by software while people keep approval over the decisions that matter. We combine rule-based automation with AI where it adds value, build in human approval, and measure return against a baseline.

Learn More
line
Enterprise AI agent platform: run AI agents with approvals, audit logs and cost limits Enterprise AI agent platform: run AI agents with approvals, audit logs and cost limits

An enterprise AI agent platform is a governed runtime where AI agents plan and carry out multi-step work through approved tools, while approvals, audit logs, access rules and cost limits keep every action accountable. Netbase offers it as a forward-looking solution, built inside your own environment rather than licensed as a product.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase