Getting into the sandbox: long-lived agent runtimes for product discovery
How we moved our research agents from an in-memory sandbox to Azure's ACA Sandboxes, and what it did for a 99-minute product discovery run.
One of the features we’ve developed at Metly is a workflow for the Value-Added Medicine (VAM) research process, which explores new versions of medicines based on known products. The aim of the VAM research process is to identify the unmet needs of patients, and identify feasible additions to existing products that can address those patient needs. To automate this research process, we have built a flexible system of agents which can process wide sections of data, and explore along various winding paths to find useful outcomes. In one run, our agents evaluated nasal medicines nearing loss of exclusivity. Within just over an hour and a half, our agents read and connected 288 sources, covering what used to be weeks of work. However, our agents need somewhere to live, and getting this right helped drop a typical run’s time by 40%.
The VAM research process needs to scan therapeutic areas for existing products, explore the capabilities of these products and then identify what limitations exist with reputable clinical evidence. Beyond just finding existing limitations, these limitations then need to be validated in both client capability and market need.
This deluge of data must be handled carefully. The data is diverse in source and form (e.g., Excel files, PDFs, web sources), and the process must be adapted, as new avenues of exploration open or close. Furthermore, each step may require both simple and well-reasoned decisions. Only industry experience separates winners from losers. LLM-powered agents bring new capabilities in data processing and linking, which is what makes them so well suited to solving this research process. Our research tooling has been developed to allow clients to use their own data safely. With this backbone custom agents can be created to explore other requirements such as internal portfolio or strategic decisions.
A Quick Reminder on Agents
An agent is remarkably similar to any experience you might have had with a chat service such as ChatGPT or Claude. An agent uses the same models but runs repeatedly toward a goal. Agents are most useful when running with tools that allow them to interact with other systems, such as reading a database, writing to a file, or communicating with other agents. A governing system of tools is referred to as an agent harness. Agent harnesses are widely available, and there is a chance you have already used one such as Claude Code or OpenAI’s Codex.
How We Use Agents At Metly
At Metly, we use AI agents to answer open-ended questions by analysing customer data in chat and in long-running, durable workflows. Each agent combines an LLM, which provides the reasoning, with a harness that manages the reasoning loop, supplies domain context, and gives the LLM access to the right tools.
We have chosen to use Pi as the core team behind the harness have stripped back the idiosyncrasies of the other harnesses to a lean and powerful machine: an agent runtime that lets the model work in a persistent workspace. Agents can run generated code to analyse Excel spreadsheets, PDFs, and Word documents, and use the file system to keep working notes outside their context window.
Because we are dealing with a mix of unstructured data and sensitive internal data, it is important that we isolate the processing that the agent does en route to its goal. So every agent works within its own sandbox. Sandboxes are isolated Unix-based runtime environments that provide the capabilities of a computer to an agent.
One of the popular solutions in enterprise AI is LangChain and associated LangGraph solution. At first read those might seem to be relatively clear off the shelf solutions. LangChain’s functional API and Deep Agents can be used to solve similar issues. But our strength in research comes from flexibility. The models decide where to explore, not any control flow written in code. The graph would reduce down to a single agent loop. Additionally the LangChain checkpoints persistence model would add another layer of storage alongside our sandbox’s existing filesystem. Our exploration has taught us an analogue to the bitter lesson: the more you are in the way of the model, the less you can achieve. The team behind Pi have ensured there is little between the harness and the model, and can easily be swapped or edited to our needs. We get our best results through careful evaluation rubrics and test runs reviewed with expertise.
Within these sandboxes, agents are provided the freedom to choose how they answer a question, while access to customer data and code execution remains authorised and secure. This gives us a huge amount of flexibility and power to process data in ways we couldn’t before. For example, an agent can load multiple sales reports, marketing registrations in different languages and then link and cross-compare in minutes what would have taken hours or days before.
Our Sandbox Requirements
To select an ephemeral workspace sandbox for our agents, we needed infrastructure that would:
- Reuse the same authorised tools and data sources as our chat agents
- Isolate authorisation and security to never allow access where an agent shouldn’t
- Support long-running research that is traceable and durable
- Support extensions to Pi or easily swap for another
- Be simple to set up in our existing infrastructure
Types of Sandboxes
There are two distinct types of sandboxing solutions we care about in the cloud: in-memory and native. In-memory sandboxes are added as a library package, e.g., a Node package, and contain an abstraction layer for your agent. Native sandboxes rely on fewer layers of abstraction, running similarly to Docker containers.
In-Memory Sandbox
We started with a lightweight in-memory sandbox library that runs inside our own servers. It was quick to set up and fitted our existing code.
Under the hood: one manager process runs many sandboxes. Each is a WebAssembly sandbox with its own Linux-like workspace: a filesystem, a shell, and a JavaScript or Python runtime. Our deployment pipeline bundles the runtime packages, the Pi harness and its configuration into one package, and the manager starts every sandbox from it, so every agent gets the same environment.
One manager could run many sandboxes on each of our servers and scaled with load, which was appealing. It took each request from a queue, started a sandbox and handed the request over. The harness inside the sandbox talked directly to our API tools and AI model proxy, and logs and traces went to Phoenix, our tracing tool, with little effort. That shared manager was also the weak point: when it got stuck, every research run on that server stopped.

Setting it up was easy: we installed one package that fitted straight into our existing TypeScript code. But VAM pushed it hard, and it became slow. A single VAM run makes over 2,000 tool calls and more than 1,000 calls to the AI model, works through lots of files, and has agents constantly passing work to each other. Every layer of abstraction added to the wait.
From our experimentation, a file read took 2.7s to complete. When reaching a checkpoint, it took 7.5 minutes to read the research artifacts. And in one 3-hour run, 85 minutes were spent reading files. Furthermore, when multiple agents simultaneously return responses to requests made from a requesting agent, the abstraction layer would encounter runtime errors, causing catastrophic failure of the entire workflow. Lastly, extensions, packages, and tools had to be abstracted through shims, further introducing latency within the process.
Near-Native Sandbox
After our first sandbox, we went back to the drawing board and evaluated near-native options such as Kata Containers. There are also paid sandbox providers, such as E2B or Daytona, but they require additional thought or contracts regarding data storage within your cloud layer. We ultimately chose Azure’s recently released ACA Sandboxes. Our workflows don’t run continuously through the week, and our use of sandboxes is sporadic so this fits well with how ACA spins up sandboxes. They also have excellent kernel-level security isolation, and it is simple for us to share access to AI services without handing the keys to the sandbox.
The move to ACA Sandboxes was simple because the security model stayed the same. We felt the benefits almost immediately: the Pi harness, its extensions, tools and packages needed no abstraction layers. Azure handles scaling and bills per second while sandboxes run, and we kept our existing queues to control how much work runs at once.
Our takeaway: in-memory sandboxes may well work for your use case, but the closer you are to running a native computing environment, the better your experience with long-running agents will be.
Lastly, for a process such as VAM, the volume of data is too large for a single agent’s context window to handle. As such, subagents are required. These subagents can either share the same sandbox or sandboxes can be unique to each subagent. The direction we’ve gone at Metly is to couple subagents to a sandbox to allow for multiple lower-resource sandboxes and contain the catastrophic failure of a single sandbox.
Security Model for Our Sandboxes
Our sandbox gives agents a runtime environment that enables flexible research. Our agents can choose how, what, and where to focus their research on, so long as it defensibly answers the user’s initial question. Although each agent has exploratory freedom within its sandbox, Metly retains control over data access, artifact persistence, and the lifetime of the agent’s session.
Agent Identity and Data Access
It is important that every agent running only has the same access as the person who requested it to run. To do this, each agent receives a short-lived token scoped to the user and/or tenant linked to the research, together with their permitted capabilities. Metly checks the token on every API tool call, so the agent can access only the data and operations authorised for that user, tenant, and agent.
Long-running executions retain the authority of the user and tenant that initiated them. This lets an agent perform authorised background work against the same data and tools available to the user, allowing the agent to automate the data interactions of the user. We also make sure to never directly hand over keys to any agents.
File Lifecycle
Research artifacts remain in the sandbox while the agent’s session is active. The persistence of research artifacts in a filesystem provides the agent a place to inspect source material, retain notes, and run intermediate analyses without placing that working state in the LLM context.
When an agent wants to store a result, it is first required to ask our API for permission. It can then only save to the location our API approves. Agents can also persist research artifacts in long-term blob storage for future use. This is initiated by the agent, to the API, which issues a signed URL to an authorised blob storage path. The use of the API tool to issue a pre-signed URL prevents agents accessing unauthorised data.
Session Termination
A sandbox exists only for the duration of its session; the sandbox is removed once the session ends. Research notes, generated code, and downloaded material are removed alongside the sandbox. Artifacts explicitly uploaded to secure long-term storage remain available for later use. This preserves the research results and trace without retaining the full agent workspace. This long-term storage remains accessible only to users whose permission was used to create the agent instance.
How We Now Use Agents for our Workflows
For complex, long-running, dynamic workflows like the VAM workflow, we’ve set up our sandbox with our agent alongside its required extensions.

For VAM research we split the work between lead and worker agents. The lead role orchestrates the workflow into phases, while the workers execute sub-tasks orchestrated by the lead. The phases are not hard coded, but guided along a rubric which scores each idea as worth exploring or dropping. This separation provides a balance between linear workflows and flexibility, where the steps orchestrated by the lead may be predictable, but the research outcome is flexible and governed by the decisions made within the worker agents.

Each worker is able to query data directly from Metly, using the authentication token tied to the workflow’s associated organisation. This allows the agents to query user data and write analysis scripts in real code, replicating the tasks an analyst would perform if manually performing this process.

Results
The result of using agents in sandboxes allows us to execute highly complex, long-running workflows. Our agents are able to process complicated data sets such as large Excel files or PDFs. The sandboxes provide agents with an environment that plays into their strengths, whilst maintaining security. Running in the ACA sandboxes significantly reduced complexity whilst speeding up our entire process. Agents can track their progress, backtrack over dead ends and update goals as new information is found. LLM agents generally are intelligent enough now that they thrive without overly restrictive instructions. Modern agents shine when given a well-defined goal and tools that are powerful enough to be used safely in pursuit of the goal.
In one such run, we evaluated nasal medicines nearing loss of exclusivity. In 99 minutes our group of agents read and connected 288 sources. They vary from EMA and national medicine registers to clinical trial records, patent filings and market data, linked together with treatment guidelines and our internal data. The agents weighed out 16 ideas of development and checked each against metrics an analyst would normally ask: what evidence do we have, what patient group needs this, how large is the market, where else is this licensed and being developed, are there any rights standing in the way and what will it take to develop? None of the final results can stand without the final team’s judgement, but weeks of groundwork are compiled in just over an hour and a half.
It should be noted that while this run took 99 minutes, our system can now sustain research over far longer horizons. The continued weakness of agents is in understanding exactly what constitutes a good goal. As of 2026, LLMs are continuously astounding the world with their progress in verifiable domains such as programming or mathematics. However, this growth in performance remains jagged, and results in non-verifiable domains such as long-form writing or general research lag behind. This is why our expertise within the pharmaceutical domain is so critical in providing valuable results.
If you have any interest in solutions to commercial challenges that can benefit from our long horizon research, you can contact us or schedule a time for us to get back to you.