Agents Are Software: Give Them a Delivery Pipeline

Published: at 01:35 AMSuggest Changes

A product manager edited a prompt at 16:50 on a Thursday and went home. Three sentences about tone in refund conversations. The change reached every customer the agent serves that evening, because the prompt lives in a configuration screen with a save button. No review. No evaluation run. No line in a change log. No obvious way back to the version that was working at lunchtime. The code repository was untouched, which is precisely why nothing in the delivery process noticed.

Harness surveyed 700 engineers and engineering leaders in organisations already working with agents, and found 75% with agents in production. Only 53% of agent-related changes travel a standard build-and-deploy pipeline before reaching users. More than a third run less than half their agent changes through one; 7% run nearly all. [Harness, September 2026]

Read that as a release-engineering problem rather than an adoption statistic. The teams on the wrong side of that split are not careless. They have a delivery pipeline, they use it for services, and they have quietly concluded it does not apply to the thing that now talks to their customers.

What you are actually releasing

An agent is not one artefact. It is an assembly: instructions and prompt templates, tool schemas and the permissions behind them, retrieval configuration and the state of the index, a model and its version, guardrail and policy rules, orchestration logic, and whatever memory the agent accumulates. Any one of those can change behaviour on its own.

An older warning about machine learning systems applies with more force here. Sculley and colleagues named it the CACE principle — changing anything changes everything — and argued that entangled configuration, not exotic algorithms, drives long-run maintenance cost. [Sculley et al., 2015] An agent carries more coupled surfaces than the models that warning was written about, and a far shorter path from edit to customer.

So the unit of release is the whole assembly, versioned together, with one identifier that production can name out loud. Harness found 42% of organisations manage agent changes through the same tooling they use for code, with workarounds, and only 34% have a configuration system built for agent behaviour. Nine per cent still need a full code redeploy to edit a prompt. [Harness]

That last number says the pipeline was never designed for this. A prompt edit requiring a redeploy gets batched, deferred, and eventually made in a hurry. A prompt edit routed through a pull request produces a green tick from reviewers who cannot see what the prompt does. Neither is a control.

The gate that has never blocked anything

Harness asked whether evaluation would catch a production-impacting failure. 74% said yes. It then asked whether a gate exists that automatically blocks a bad release, with no exceptions. 19% had one. Among organisations confident in their evals, roughly four in five did not. [Harness]

VentureBeat’s Pulse research shows the consequence. Half of the enterprises surveyed had deployed an agent or model feature that passed internal evaluation and then caused a customer-facing failure; a quarter had seen it happen more than once. Only 5% fully trust automated evaluation, with poor alignment to real-world outcomes the most cited reason. Two-thirds already permit, or are engineering toward, deploying low-risk agent changes with no human in the loop. [VentureBeat Pulse Research, June 2026]

I have sat in the review where an agent was cleared for wider release on eleven passing test cases. Nobody in the room asked how many times each case had been run. Anthropic’s evaluation guidance addresses exactly that gap: a system that succeeds at least once across repeated attempts is a different claim from one that succeeds every time, and only the second claim supports a release decision. Vary the phrasing, run the same scenario repeatedly, break a tool deliberately, then check whether the business outcome still holds. [Anthropic, January 2026]

A promotion gate needs three things most organisations have not written down. A fixed standard applied to every change: Harness found 58% check each promotion against a repeatable standard, which leaves 42% deciding case by case, including 11% with no formal process at all. A named approver for each class of change, because “the team reviewed it” is not an accountable decision. And a record of what the gate stopped. A gate that has never blocked a release is either perfect or decorative, and you should know which one you own.

Ship to a slice, not to everyone

The techniques that limit blast radius during a release are the ones least used for agents. Manual approval gates run at 42% and automated rollback at 39% — both all-or-nothing instruments. Canary deployment sits at 26% and blue/green at 25%. [Harness]

The industry spent a decade teaching itself to release progressively. It has not carried the habit across to the artefacts that behave unpredictably. Singapore’s agentic AI guidance makes the expectation explicit: roll agents out gradually, limited to certain users or features first, with continuous monitoring and testing alongside. It also asks for formal change-management and version-control processes across the lifecycle, on the reasoning that in an interconnected system a small change to one agent can cascade into failure elsewhere. [IMDA and AI Verify Foundation, May 2026]

Progressive delivery for an agent has more shapes than a traffic split. A hold-back cohort. A single task class enabled first. Read-only tool scope before write access. Shadow execution that produces the decision without acting on it. Each limits what a bad release can reach while the team is still learning what a bad release looks like.

One caution. Canary on a non-deterministic system is not canary on a service, because one clean run proves an agent can do the task rather than that it will. Sample size and a quality signal both matter. VentureBeat found roughly half of respondents monitoring whether the agent was functioning — latency, errors, cost — against about a quarter automatically checking whether its output was correct. [VentureBeat Pulse Research] A canary watching only uptime will promote a confidently wrong agent without hesitation.

Name the owner, and test the way back

Two release artefacts get skipped because they feel like operational detail. They are not.

The first is ownership. Ask which system holds the authoritative list of agents running in your environment, and who is named against each entry. Harness found a centralised agent inventory leading at just 30%, with 13% using an internal developer portal, 12% a custom list, and 8% still tracking ownership in a repository README. [Harness] An agent with no owner has nobody to promote its next change and nobody to call at 03:00.

The second is the way back. 76% of organisations believe they could disable a misbehaving agent within fifteen minutes; 33% have an instant kill switch in production and 39% automated rollback. [Harness] OWASP’s agentic risk work lists cascading failures and rogue agents as standing categories and points at the same controls: circuit breakers, validation between plan and execution, rollback mechanisms, kill switches, drift monitoring. [OWASP GenAI Security Project, December 2025]

A kill switch is a design decision taken at release time, not an incident-time improvisation. It needs a defined blast radius — which agents, which tools, which queues — an invocation path that does not depend on the same identity provider or ticketing system the agent relies on, a named decision right, and a rehearsal. Choosing between four ways to stop an agent while customers are waiting is how a fifteen-minute target becomes a two-hour one.

The tooling fragmentation is measurable. Harness found the leading approach, AI-platform-native tooling, covers 24% of the market; only 9% of organisations have an integrated control plane spanning the lifecycle; and 13% describe their approach as mostly manual or a set of disconnected point tools. 84% see high or moderate value in a single control plane, and 76% expect to be investing within six months. [Harness] That is a buying window, and the teams that spend it well will have written down what they need before the vendor conversation starts: versioned configuration, an evaluation gate with a named approver, progressive rollout, an inventory with owners, and a tested kill switch.

Let production write the next release

The clearest sign that a delivery pipeline exists is that it changes after an incident. Every production failure should become a permanent case in the pre-deployment suite: the escalation, the failed tool call, the wrong field update, the approval that should have been required. Anthropic’s guidance makes the same argument about converting production failures into regression tests, and OWASP’s mitigation themes for memory poisoning and cascading failures both assume the case can be replayed.

Then instrument the release process itself. Harness found 58% of organisations reporting more production incidents per 100 changes since deploying agents, against 25% reporting fewer. [Harness] That ratio is a fitness function for the pipeline, and the number a platform team should own. If it climbs while agent change volume grows, the pipeline is not keeping pace with the change surface, and the answer is rarely more reviewers.

The test

Take the agent that changes most often. Ask its owner to show you the last three changes: what moved, who approved each one, against which written standard, what the gate would have stopped, which version is running now, and how they would reverse it inside five minutes.

If the answer involves a screenshot, a chat thread, or a prompt that is “in the repository somewhere”, you have a deployment habit rather than a delivery pipeline. The remedy is not a procurement decision. Version the assembly, write the promotion standard down, release to a slice, and name the person who owns the kill switch before you need it.

Sources

Next Post
When the Agent Goes Down, Who Can Still Do the Work?