AI Agent Sandbox: We Shipped the Bug First

TL;DR: We shipped a security advisory before we shipped a good AI agent sandbox. Untrusted Python was running in our backend process, and the fix taught me more about this topic than any of the reading I had done beforehand. This is what I learned, what happened when I finally wrote a script to ask our own sandbox what it could see, and why the answer changed how I build workflow steps. There is a runnable workflow at the end.

Key Takeaways:

• Our own CVE came from two code paths that should have been identical, where the second one arrived later, looked smaller, and nobody re-asked the question
• Twenty minutes of probing your own container beats any amount of reading vendor documentation about isolation
• The researchers who tested this found that what gets exploited is almost always misconfiguration, not the container technology itself
• A sandbox controls what a step can reach and has no opinion on whether the step should have happened, which is a separate control and a separate mechanism
• Running code in a throwaway container cost about a third of a second, so the performance objection is much weaker than the discussion suggests

Table of Contents

• The Bug We Shipped
• What I Did Differently Next
• Twenty Minutes That Settled It
• What the Researchers Actually Found
• The Question I Now Ask First
• A Sandbox Is Not a Yes
• The Workflow
• What It Costs
• Where I Would Not Trust This
• FAQ
• What Is Heym?

The Bug We Shipped

This is written for people who run their own automation infrastructure, on their own hardware, and who have at some point wondered whether the isolation they configured is real. It is about running code safely as a step in a workflow. It is not a guide to choosing a hosted sandbox vendor, and it does not cover multi-tenant platforms serving untrusted strangers at scale, which is a harder problem than the one I am describing.

In July we published CVE-2026-67543. High severity, 8.8. The summary is one sentence and it is not a flattering one:

Unsandboxed skill code execution: skills run via a local backend subprocess without the Docker isolation that Python tools mandate

Definition: An AI agent sandbox is an isolated environment where code an agent runs cannot reach the system that started it. Usually a container created for one execution and destroyed afterwards, configured so the code cannot read the host's secrets, write to its filesystem, or reach the network unless explicitly allowed.

In short: Python bundled inside an agent skill ran in our backend process rather than in the container our Python tools had always been required to use. I wrote up that advisory in detail when we shipped the fix, including the full exposure list and what it means that skills travel inside importable templates, so I am not going to repeat it here. If you want the post-mortem, it is in agent skills under the section with the same uncomfortable title.

What I want to talk about is the part that piece does not cover, because it had not happened yet.

What I Did Differently Next

Two days ago I shipped a Code node. It runs Python you write as a step in a workflow, and it installs whatever packages that Python needs, from PyPI, at run time.

So this is the third path in the same product that executes code somebody else wrote, and by some distance the most exposed. An agent tool is a function an operator authored. A skill is a bundled script. A Code node execution pulls in a dependency tree that neither of us has read, from an index anyone can publish to, on every single run.

Having already shipped one path that quietly ran untrusted Python in the wrong place, I was not interested in reasoning my way to a third answer. I wanted to know rather than believe. Which is how I ended up doing the thing that turned out to be the actual lesson.

Twenty Minutes That Settled It

What that node needed is a python code execution sandbox in the plainest sense: your code, your requirements.txt, a container that exists for exactly one run and is then destroyed.

I built the isolation the way you would expect. Fresh container per execution, destroyed afterwards, no privileges, no secrets. And then I did something I had never done before on any of the previous paths, which in hindsight is the actual lesson of this whole post.

Asking instead of assuming

I wrote a script whose only job is to look around and report what it can see, and I ran it through the real execution path. Not a test. The same function the product calls.

Ten rows. Twenty minutes of work. And it converted a page of adjectives I had written in the documentation into something I could actually stand behind.

I want to be honest about why this felt significant, because on paper it is unremarkable. Every one of those answers is what I had intended when I wrote the configuration. The point is that until I ran it, intending was all I had. The advisory in July was also a case where everyone intended the right thing.

The privilege nobody uses

There is a second reason this matters, and it is specific to running things yourself. If you use a hosted platform, you cannot do this. You get the vendor's description of their isolation and no way to check it, which is not their fault, it is just what buying infrastructure means. When you run a self hosted ai workflow automation stack, the box is yours and the check is available to you. That is the actual privilege of self-hosting, and almost nobody uses it.

What the Researchers Actually Found

I went looking for someone who had tested this properly rather than asserted it, and the answer was better than I expected.

In March 2026 a team at the UK AI Security Institute published SandboxEscapeBench, which measures whether a language model can break out of a container. It was accepted as an oral at ICML 2026. They built it as a capture the flag exercise with a sandbox inside a sandbox, so they could ask a genuinely dangerous question safely, and they modelled the attacker as an agent that already has shell access inside your container. Which is exactly the situation you are in on a bad day.

Their own summary contains the line I keep coming back to:

misconfigurations like exposed Docker sockets, excessive privileges, and unpatched runtimes can be reliably exploited by frontier models

Across eighteen escape scenarios, the pattern was that advanced models could reliably get out of common misconfigurations, and none of them solved the hardest scenarios.

Key principle: The container is rarely the thing that fails. The configuration is. That means isolation is not something you buy once, it is something you can check, and checking takes minutes.

Read that twice, because it is more reassuring than the headline suggests and more demanding at the same time. Containers are not made of paper. What is reliably exploitable is the set of mistakes people make while setting them up. So "is a container enough" has no answer. The answerable question is whether your container, started the way you started it, is enough.

Three doors, and only two are free

Their three items are a gift, because they turn an anxiety about your AI agent sandbox into a checklist:

That third row is the honest one. A project can ship good defaults for the first two and they will hold no matter what you do. It cannot patch your kernel. If you run an ai agent sandbox docker setup on a host you last updated a year ago, the first two rows are still green and you still have a problem.

The Question I Now Ask First

Here is what changed in how I think, and it is the part I have not seen written down anywhere.

On a hosted platform, isolation is a property of the product. You cannot weaken it because you do not have the controls. On your own hardware, isolation is a property of configuration, and configuration has switches. So the question is not which technology you chose. It is: which parts of this system will cheerfully run untrusted code without a sandbox if a setting says so, and who can change that setting?

Why I removed the escape hatch

Almost every project has an escape hatch, for honest reasons. Docker is not always running on a laptop. Contributors need things to work. A fallback is the pragmatic answer, and I am not against them.

The danger is that a single switch usually covers more paths than the person flipping it realises.

So when I built the Code node I broke the pattern deliberately. Our agent tools and skills share a setting that can select a weaker local mode, which is fine, because those run Python that an operator wrote. The Code node ignores it entirely. There is no setting. Without Docker it refuses to run and tells you why.

The reason is narrow rather than ideological. It is the only path that installs arbitrary packages from the internet at run time, which is a supply chain and not just a script. It is also the layer a prompt injection eventually cashes out in, since an attack that talks an agent into running something is only as damaging as the surface it runs on. When both the code and everything it depends on are untrusted, a weaker local mode is not a boundary, and offering a switch that implies otherwise would be worse than offering nothing.

That decision has a real cost, and I would rather name it than pretend. Anyone running an ai agent local sandbox on their laptop has to start Docker before that step works. That is friction in a development loop, and friction is a genuine tax. I took it because the alternative failure is silent, and I had already lived through one silent failure in July.

A Sandbox Is Not a Yes

Everything so far is about one question: what can this step reach. It is a good question, it has a measurable answer, and it gets nearly all of the attention.

It is also the narrowest of three questions, and the other two are strangely absent from the conversation. I read through the pages that rank for this topic. The ones about sandboxes do not mention approvals or audit trails. The ones about AI governance and approvals do not mention execution isolation. Different authors, different buyers, and they never reference each other.

That gap produces a very specific and very common failure: a system with excellent isolation that confidently does the wrong thing.

Picture a workflow that reads support tickets, works out which ones deserve a refund, calculates the amount, and issues it. This is the scenario where an AI agent sandbox gives you the most false comfort. Now suppose the calculation is wrong and it refunds two hundred people ten times what they were owed. Walk that through the isolation checklist. No privilege was used. Nothing was written. Nothing escaped. The sandbox did its job perfectly and you are still having the worst Monday of your career.

Notable fact: A sandbox breach is loud and rare. A correctly executed wrong action is quiet and common. Most teams buy protection against the first and ship without protection against the second.

The control that catches the refund is a human checkpoint on the consequential step, and the design question is where those go. The honest answer is fewer places than instinct suggests, because gating everything produces a reviewer who approves without reading, which manufactures an audit trail of rubber stamps. We went into that trade in human in the loop AI agents, including the research showing that past a point more gates make things worse. The rule that survives contact with reality is simple: gate the action, not the arithmetic.

The third question is evidence, and it is worth one practical note. A Code node returns what the code gave back, what it printed, and what happened during dependency installation as three separate fields. That third one earns its place more often than you would think. When something that worked yesterday breaks today and nobody changed the code, the answer is usually that a dependency resolved differently, and without that field you will spend an afternoon debugging the wrong layer. What else is worth keeping from a run is the subject of AI agent observability.

The Workflow

Enough theory. Here is the shape as something you can run.

This one takes a list of expenses, finds the outliers with real Python in a sandboxed step, and then, only if something was flagged, hands off to an agent that drafts a note and pauses for a human to approve it. Nothing goes out without a person saying yes. The maths is not gated, because gating maths is how you train reviewers to click approve.

{
  "heym": true,
  "nodes": [
    {
      "id": "spend_note",
      "type": "sticky",
      "position": {
        "x": 40,
        "y": 40
      },
      "data": {
        "label": "setupNote",
        "stickyTitle": "How this is protected",
        "stickyColor": "violet",
        "stickyWidth": 320,
        "stickyHeight": 230,
        "note": "FlagOutliers runs in a throwaway container: non-root, read-only filesystem, no Docker socket, no backend secrets, and Allow network off.\n\nDraftTheNotice has human review on, so nothing is sent until a person approves it."
      }
    },
    {
      "id": "spend_input",
      "type": "textInput",
      "position": {
        "x": 40,
        "y": 330
      },
      "data": {
        "label": "expenseRows",
        "value": "[\n  {\"vendor\": \"Cloud Host\", \"amount\": 412.50},\n  {\"vendor\": \"Design Tool\", \"amount\": 96.00},\n  {\"vendor\": \"Data Vendor\", \"amount\": 388.20},\n  {\"vendor\": \"Unknown LLC\", \"amount\": 9840.00},\n  {\"vendor\": \"Coffee Service\", \"amount\": 143.75}\n]",
        "inputFields": [
          {
            "key": "text"
          }
        ]
      }
    },
    {
      "id": "spend_flag",
      "type": "code",
      "position": {
        "x": 400,
        "y": 330
      },
      "data": {
        "label": "FlagOutliers",
        "codeSource": "import json\nimport statistics\n\n\ndef main(params):\n    rows = json.loads(params.raw)\n    amounts = [r[\"amount\"] for r in rows]\n    median = statistics.median(amounts)\n    limit = median * 3\n    flagged = [r for r in rows if r[\"amount\"] > limit]\n    return {\n        \"median\": median,\n        \"limit\": round(limit, 2),\n        \"flagged\": flagged,\n        \"flagged_count\": len(flagged),\n    }\n",
        "codeRequirements": "",
        "codeParameters": "{\n  \"raw\": \"$expenseRows.text\"\n}",
        "codeAllowNetwork": false
      }
    },
    {
      "id": "spend_gate",
      "type": "condition",
      "position": {
        "x": 760,
        "y": 330
      },
      "data": {
        "label": "AnythingToReview",
        "condition": "$FlagOutliers.result.flagged_count >= 1"
      }
    },
    {
      "id": "spend_agent",
      "type": "agent",
      "position": {
        "x": 1100,
        "y": 190
      },
      "data": {
        "label": "DraftTheNotice",
        "model": "gpt-5.5",
        "hitlEnabled": true,
        "systemInstruction": "You write a short internal note about flagged expenses. State the vendor, the amount, and the threshold it exceeded. Do not speculate about intent. Before sending anything, request human review and wait for approval.",
        "userMessage": "Flagged rows: $FlagOutliers.result.flagged and the threshold was $FlagOutliers.result.limit"
      }
    },
    {
      "id": "spend_clear",
      "type": "output",
      "position": {
        "x": 1100,
        "y": 470
      },
      "data": {
        "label": "NothingUnusual",
        "outputSchema": [
          {
            "key": "median",
            "value": "$FlagOutliers.result.median"
          },
          {
            "key": "checked",
            "value": "true"
          }
        ]
      }
    }
  ],
  "edges": [
    {
      "id": "spend_e1",
      "source": "spend_input",
      "target": "spend_flag"
    },
    {
      "id": "spend_e2",
      "source": "spend_flag",
      "target": "spend_gate"
    },
    {
      "id": "spend_e3",
      "source": "spend_gate",
      "target": "spend_agent",
      "sourceHandle": "true"
    },
    {
      "id": "spend_e4",
      "source": "spend_gate",
      "target": "spend_clear",
      "sourceHandle": "false"
    }
  ]
}

Three things in that file are the whole argument for how an AI agent sandbox should sit inside a workflow. codeAllowNetwork is off, so the step doing the calculation cannot phone anywhere even if the logic is wrong. hitlEnabled is on, so the step that talks to humans waits for one. And the split between them is the point: the container protects the calculation, and the gate protects the consequence.

Swap expenseRows for an HTTP step against your real system and it becomes something you would actually use. Fetch with HTTP, pass the body in as a parameter, and leave the code step with no network at all. That is the pattern I would default to for almost everything.

If you would rather start from something already published, Python Latency SLA Report is the other Code node template. It normalises messy timestamps with a real date parser and computes p50, p95 and p99, which is a good illustration of when to reach for code at all. Percentiles cannot be expressed in a field mapper, and a model would give you a slightly different answer every run, which for an SLA report is not a quirk but a defect. There are more of these in our AI workflow automation examples.

What It Costs

The discussion around this topic is preoccupied with cold starts. Vendors sell warm pools and standby capacity.

We went the other way and cache nothing at all. No warm pool, no kept dependencies, nothing survives a run. That is the slowest possible choice, which makes it a fair thing to measure.

I ran the dependency case twice to make sure I was not reading a warm cache, and got 1.24s then 1.00s. Since nothing is kept between runs, that spread is ordinary variance rather than the second run benefiting from the first.

Two things follow. A third of a second for create, run and destroy makes the performance objection much weaker than the marketing around it suggests, at least for workflow steps. And the cost is dependencies, not containers. Leaving the requirements empty is roughly three times faster than installing one small package, which is a far bigger lever than anything about the runtime.

If you are thinking about the overall bill rather than one step, AI agent cost optimization is the relevant read, and the headline there applies here: model calls dominate, and a second of container time is noise next to one unnecessary round trip to an LLM.

Where I Would Not Trust This

A post arguing for measurement owes you the limits of its own measurements, so briefly. There are three places I would not present our AI agent sandbox as the right answer.

A well configured container is not a virtual machine. Some products in this space use Firecracker or gVisor, and those are stronger boundaries, particularly against attacks aimed at the kernel. LangChain's walk through of the trade-offs is the fairest summary I have read of when the extra isolation is worth its cost. The AISI results support taking that seriously, since the larger models did succeed at the middle difficulty scenarios. My position is that a properly configured container is a real boundary for the threat this step faces, which is a bad dependency rather than a determined attacker with a kernel exploit. If you are running genuinely hostile code for strangers, you want a stronger boundary than we ship, and you should believe that rather than me.

I should also say that a code execution sandbox open source contributors can read is not automatically safer than a managed service with a security team behind it. What open source gives you here is the ability to run the check yourself, which is worth a great deal, and the responsibility for the patching that nobody can do on your behalf.

And the check proves the configuration took effect. It does not prove there are no bugs. The July advisory was not a misconfiguration, it was a path nobody thought to protect, and no probe of this kind would have found it. That is what tests and a security process are for.

The obvious next step, and the one I intend to take, is to stop citing SandboxEscapeBench and run it. It is open, and a self-reported score would be worth more than anything I have written here. My guess is that we pass the misconfiguration scenarios comfortably and that the kernel-layer ones are where a container-based approach shows its limits, but a guess is exactly what this whole post argues against. When we run it, the number goes here, whichever way it lands.

I expect this area to move quickly in one specific direction. Right now isolation is sold as a feature and verified by nobody. As agents get better at exactly the exploitation the AISI paper measures, I think buyers start asking vendors for evidence rather than architecture diagrams, and the vendors who can produce it will be the ones who let you look inside.

FAQ

What is an AI agent sandbox?

An AI agent sandbox is an isolated environment where code your agent runs cannot reach the system that started it. In practice it is a container that gets created for one execution and destroyed afterwards, configured so the code inside cannot read your secrets, cannot write to the filesystem, and cannot call out to the network unless you allowed it. The thing people get wrong is treating it as a product you buy. It is a set of choices you make, and the same container technology can be a real boundary or a decoration depending on those choices.

Is Docker enough to sandbox an AI agent?

A container you started without thinking about it is not enough. A container you configured deliberately usually is. The UK AI Security Institute tested this and found that frontier models can reliably exploit exposed Docker sockets, excessive privileges, and unpatched runtimes. Notice that none of those are flaws in Docker. They are things a person configured. A container running as root with the Docker socket mounted is a shell on your host wearing a costume, and the same container started as a non-root user with no socket and a read-only filesystem is a genuine wall.

How do I know my sandbox is actually working?

Run code inside it that reports on its own surroundings, and read what comes back. Have it print the user it is running as, whether it can see the Docker socket, which of your secrets are visible in its environment, whether it can write a file, and whether it can open a connection to the outside world. This takes about twenty minutes to write and it is the only way to know that the configuration you think you deployed is the configuration that is actually running. Nothing else you do in this area has a better return.

Is it slow to run code in a fresh container every time?

Much less than people expect. A full cycle of starting a throwaway container, running Python in it, getting the result back, and destroying the container took about a third of a second when the code had no dependencies to install. Adding one package and installing it completely fresh, with nothing cached, took it to roughly a second. Dependencies are the cost, not containers, which is the opposite of what most of the discussion assumes.

Does a sandbox mean my AI workflow is safe?

No, and this is the most expensive misunderstanding in the area. A sandbox controls what a step can reach. It has no opinion about whether the step should have happened. If your workflow calculates a refund incorrectly and issues it, nothing escaped anything and the outcome is still a bad day. Containment and authorization are different problems. You need a sandbox for the first and a human approval gate for the second, and a trace afterwards so you can tell what actually ran.

Should the language model write the code that runs in the sandbox?

It depends on whether you need the same answer twice. If a model writes the logic fresh on every run, you get a slightly different program each time and the sandbox is your only defence against that. If a person writes it once and it is saved as part of the workflow, the run is repeatable, you can read it in version history, and the sandbox is protecting you against something else entirely, which is that the workflow might have come from someone you do not know. Both are reasonable. Not noticing which one you are doing is where people get hurt.

What Is Heym?

Heym is an open source, AI-native workflow automation platform you can self-host. You build workflows on a visual canvas from typed nodes, including agents, LLM steps, integrations, and a Code node that runs your Python in a disposable container. It supports human approval checkpoints, execution traces, and OpenTelemetry, so the three controls in this post are configuration rather than custom engineering.

If you take one thing from this, make it the twenty minutes.

I had read a great deal about isolation before I wrote any of it, and none of that reading told me as much as one short script that reported back what our own container could see. It is the cheapest security work available to anyone running their own infrastructure, it needs no budget and no approval, and it is the only way to find out whether what you deployed is what is running.

Every ai agent sandbox open source or otherwise is a set of choices somebody made. Go and read yours, and if you would rather start from something that already has the choices made, our self-hosted platform ships with them on by default.