The past year has seen an extraordinary rise in AI agent frameworks. Much of the industry conversation has understandably focused on areas such as reasoning, planning, memory, and tool usage. These capabilities are important because they determine what an agent can do. Yet while experimenting with agentic systems, I found myself drawn to a different question altogether, one that enterprise architects have wrestled with for decades:
Where should state actually live?
This question is not new. Long before AI agents became mainstream, architects designing distributed applications grappled with how to separate computation from persistence, execution from storage, and transient workloads from durable business data. As I worked through a recent multi-agent experiment, I began to realize that many of the lessons we learned in cloud computing and distributed systems may be reappearing in the world of AI, albeit in a different form.
The insight emerged during a customer support project I built as an experiment. My initial idea was relatively simple. Instead of keeping specialist agents running continuously, I would create an agent only when a request arrived, allow it to complete a specific task, and then terminate it immediately afterward. In my mind, the interesting aspect of the architecture was the notion of an ephemeral agent. By the end of the project, however, I realized I had been focused on the wrong thing.
The more important realization was that execution should be ephemeral, but state should not.
At first glance, this seems like a subtle distinction. In practice, I believe it has significant implications for how enterprise AI systems should be designed and governed.
The Problem with Treating Agents Like Employees
Many multi-agent architectures naturally evolve into something that resembles an organizational chart. A system might contain a Billing Agent, a Support Agent, a Refund Agent, and a Research Agent, each continuously available and waiting for work. These agents often maintain access to tools, credentials, and conversation histories over extended periods of time.
There is nothing inherently wrong with this approach. In fact, it works extremely well for many use cases. However, when viewed through an enterprise lens, it introduces a familiar set of concerns. Who owns long-lived credentials? How should memory be governed? What happens when an agent accumulates months of context? How do we audit decision-making across persistent entities? And perhaps most importantly, how do we reduce risk when something goes wrong?
What struck me while building the system was that these are not fundamentally AI problems. They are governance, security, and distributed-systems problems that enterprises have already encountered many times before. AI simply provides a new context in which those challenges reappear.
The Experiment
To explore these ideas, I built a customer support platform using the Bitext support dataset. The dataset contains a broad range of customer service interactions, which I grouped into twenty-seven intents spanning six major support domains: orders, refunds, shipping, billing, account management, and general support.
The architecture consisted of an orchestrator and a collection of specialist agents. Rather than maintaining a permanently running pool of agents, the orchestrator classified each incoming request and instantiated a specialist agent on demand. That agent received only the tools and permissions necessary for the task at hand. After completing the request, the agent was immediately terminated.
The first iteration of the system was far from ideal. Agents frequently exceeded execution limits, and in many cases tool outputs were not being sufficiently incorporated into the final responses. After several rounds of tuning, performance improved significantly. Task completion rates exceeded ninety-three percent, while response times dropped from approximately twenty-five seconds to just over one second.
The performance improvements were encouraging, but they were not the most important result of the experiment.
The real challenge emerged elsewhere.
The Hardest Problem Was Continuity
Consider a simple customer interaction. A user asks for the status of an order. A few minutes later, the same user asks whether the shipping address for that order can be updated.
In a traditional system, a persistent agent could simply remember the earlier conversation and continue seamlessly. In my architecture, however, the original agent no longer existed. By design, it had been terminated immediately after completing its assigned task. When the second request arrived, an entirely new agent was created to handle it.
Initially, my instinct was to extend the lifetime of the agents so they could preserve context across interactions. The more I explored that option, the more it felt like a step backward. Keeping agents alive longer meant retaining memory longer, maintaining credentials longer, and expanding the attack surface of the system. In effect, I was recreating the exact conditions I had been trying to eliminate.
The breakthrough came when I shifted my thinking away from the agents themselves and toward the platform. Instead of embedding continuity inside an individual agent, I moved continuity into the orchestration layer. Conversation history became part of the platform’s state. Every newly instantiated agent received only the relevant context required to perform its task. Once execution was complete, the agent disappeared, but the conversation persisted.
The platform remembered. The worker did not.
That distinction fundamentally changed how I thought about agent architectures.
Separating Execution from State
After completing the experiment, I stopped viewing agents as persistent intelligent workers. Instead, I began viewing them as disposable execution units operating on durable enterprise state.
In this model, the platform becomes responsible for maintaining business context. It owns identity, conversation history, audit logs, governance controls, and orchestration. The agent itself becomes responsible for only one thing: executing the current task.
This separation creates a cleaner architecture because each layer has a clear purpose. Business state remains governed, auditable, and persistent. Intelligence becomes transient, created only when work needs to be performed.
Interestingly, this architectural pattern feels very familiar. Enterprise computing has been steadily moving in this direction for years. Web servers evolved toward stateless architectures. Applications were decomposed into microservices. Serverless computing separated execution from infrastructure ownership. AI agents may simply represent the next step in that progression.
Why This Matters for Enterprise AI
A significant portion of the enterprise AI discussion focuses on models. Organizations compare benchmarks, evaluate reasoning capabilities, and debate which foundation model is most capable. While those conversations are important, they often overlook a more practical reality: most deployment challenges have relatively little to do with model quality.
In practice, enterprise adoption is frequently constrained by governance, security, compliance, and operational complexity.
An architecture built around short-lived execution aligns naturally with these concerns. Temporary agents operate under the principle of least privilege because they only receive the permissions required for the current task. Credentials exist only for the duration of an execution rather than remaining active indefinitely. Auditability becomes significantly cleaner because every request has a clearly defined lifecycle, complete with its own inputs, outputs, tool calls, and execution logs.
Perhaps most importantly, business state remains where enterprises expect it to reside: inside enterprise systems of record rather than inside an agent’s memory. Customer information remains in CRM platforms. Transactional data remains in operational systems. Policies remain within governed repositories. The agent becomes a processor rather than a custodian of enterprise knowledge.
This shift may not make agents inherently smarter, but it does make systems significantly easier to govern.
The Challenges of Ephemerality
Of course, ephemeral agents are not a universal solution. Every architectural decision introduces trade-offs.
Creating a fresh agent for every request introduces overhead. Context must be assembled, credentials provisioned, and relevant history retrieved before execution begins. These costs may be small individually, but they become meaningful at enterprise scale.
State management also becomes critically important. During my experiment, conversation history was maintained in memory. That works for a prototype but would not be sufficient for a production environment. Durable state stores, shared memory architectures, and distributed persistence layers quickly become essential.
There are practical considerations as well. Determining how much history to provide to a newly instantiated agent requires careful judgment. Too little context can result in poor decisions, while too much context increases cost, latency, and complexity. Long-running tasks present another challenge because not all work fits neatly into short execution windows.
None of these issues invalidate the pattern. They simply highlight that complexity never disappears. It merely moves. In this case, complexity moves away from individual agents and into the orchestration platform that governs them.
Not Every Use Case Should Be Ephemeral
It is important to recognize that this pattern is not universally applicable.
Some forms of AI interaction derive their value from long-term continuity. A coding partner benefits from understanding months of architectural decisions. A research assistant improves when it remembers previous investigations. A personal AI companion becomes more valuable as it accumulates knowledge about preferences and behavior.
Enterprise operational work is different. Customer support, claims processing, order management, approvals, and compliance reviews are generally discrete, task-oriented activities. They operate within clearly defined business processes and are already governed by enterprise systems. Separating execution from state aligns naturally with these characteristics.
For these types of workflows, ephemeral execution is not merely feasible. It may actually be preferable.
The Bigger Lesson
Looking back, I no longer think of this project as an experiment involving ephemeral agents. It was really an experiment about something far more fundamental.
Execution lifetime and business state do not need to be coupled.
This principle has shaped many of the most successful architectural shifts of the last two decades. Stateless services, cloud-native applications, microservices, and serverless computing all emerged from the realization that execution and persistence are distinct concerns.
Agentic AI may be heading toward the same conclusion.
The future of enterprise AI may not resemble a company filled with digital employees carrying permanent memories and standing access to enterprise systems. Instead, it may resemble a platform that continuously hires highly specialized contractors, provides exactly the context and permissions required to perform a specific task, and then releases them the moment the work is complete.
That architecture feels more governable, more secure, and ultimately more aligned with how enterprises manage risk and scale.
And perhaps that is the most important lesson of all: the future of enterprise AI may depend less on creating agents that remember everything and more on building platforms that remember what matters.
