AI Agent Governance: A Count Is Not a Control

TL;DR: AI agent governance is the set of records and controls that make an autonomous agent accountable: a named human owner, a written purpose, an enforced permission boundary, and a shutdown procedure someone can execute. Most writing on the subject answers a different question, how many agents do we have, and that number is close to useless on its own. We ran the four questions against our own platform and published what we found: 28 of 62 node types can write, send, or delete in a system outside Heym, exactly one of them requires human approval and it is off by default, and 13 of our 48 API routers emit an audit line.

Key Takeaways:

• Agent count is an inventory metric. Owner, purpose, boundary and shutdown coverage are the governance metrics, and they are the ones almost nobody publishes
• Of our 62 node types, 28 can act on a system outside the platform. One of those 28, the Code node, ships with its network off by default. The other 27 have no equivalent switch
• Human approval exists on one node type in our product and defaults to off, and the policy that drives it is prose matched by 12 positive and 12 negative regular expressions
• An audit trail is the only part of governance that survives contact with an incident, and ours covers 13 of 48 API routers, which is 27 percent
• Local inference removes the prompt from the network without removing the agent's reach, and it deletes the outbound API call that many teams were quietly using as their agent inventory

Table of Contents

• What AI Agent Governance Actually Means
• The Metric Everyone Reports
• The Four Questions
• We Ran the Four Questions on Our Own Platform
• The Audit Trail Is Where Governance Becomes Checkable
• Local Inference Does Not Remove the Identity Problem
• An Agent Register You Can Actually Run
• What to Do This Week
• What This Post Is Not
• Frequently Asked Questions

What AI Agent Governance Actually Means

AI agent governance is the set of records and controls that make an autonomous agent accountable: a named human owner, a written purpose, an enforced permission boundary, and a shutdown procedure someone can actually execute. Everything else in the category is decoration on those four.

This post is written for the engineer who wires the agent to the API, not for the person buying a control plane. If you are preparing an audit committee deck, Microsoft's Cloud Adoption Framework guidance on governing agents and the OWASP Gen AI Security Project's State of Agentic AI Security and Governance are better starting points than anything I am about to write. What I have that they do not is a system I can measure and permission to publish the numbers, including the unflattering ones.

The reason the category exists at all is that model governance was built for a different object. Content filtering, acceptable use policies and bias review were designed around systems that produce text. An agent produces consequences. It reads your email, queries your database, pushes a commit, sends the invoice. The NIST AI Risk Management Framework already asks for traceability, which is the right instinct, but tracing a model output and tracing an action taken in someone else's system are different engineering problems.

There is a distinction worth stealing from a thread on r/AI_Agents, phrased better there than in most vendor material: saying an agent should not do X is governance, and blocking the tool call before X executes is control. Most programs have a lot of the first and very little of the second. ARMO makes the same split more formally in their five-rung enforceability ladder, which is the strongest piece of writing on this subject I found and which I am not going to reproduce here. Their ladder answers how do you prove enforcement held. My four questions answer something earlier and more boring: do you even know what this thing is.

The Metric Everyone Reports

Ask a platform team how their agent program is going and you will usually get a count. Fourteen agents in production. Two hundred copilots registered. The number goes in the board deck and everybody nods.

The count is not wrong, it is just the wrong unit. AI agent sprawl is not the condition of having many agents. It is the condition of having more agents than you have records about agents. A team running forty agents with forty owner records is in far better shape than a team running twelve with none, even though twelve sounds tidier in a meeting.

I like coverage as the replacement metric because it is a percentage and percentages embarrass people into action in a way that raw counts do not. Pick the four questions, answer them for every running agent, and report the fraction you could answer. The first time you run it the number will be bad. That is the point of running it.

The Four Questions

Here is the whole framework. It fits on an index card, which is deliberate, because an ai agent governance framework that requires a workshop to explain will not be applied to the agent someone ships on a Friday.

Every ai agent governance framework I have read ends up at some version of these four, usually with more words and an extra column. Two of these are records and two are controls. Owner and purpose are things you write down. Boundary and shutdown are things the system does. Programs fail in a predictable way: the records get built because they are easy and visible, the controls get deferred because they require touching the execution path.

We Ran the Four Questions on Our Own Platform

Heym is our own product, so read this section as interested rather than neutral. Everything below was counted on 2026-08-25 against our repository at commit c2a8b6b7, and the counting method is stated in each case so you can disagree with it.

Question One: Who Owns It

This one we pass, and it is worth saying why, because the reason is not virtue. We were forced into it by a different problem.

Credentials in Heym have a single owner and an explicit sharing model. A collaborator sees that a credential is configured but never sees its value, because the response schema masks it and the write path rejects a collaborator update. That second half matters more than it sounds. Masking a value in a read without guarding the write is a well-known way to lose data: the collaborator opens the editor, the field renders empty because it was masked, autosave fires, and the owner's credential is now an empty string.

Capability secrets are stored as SHA-256 digests through a single shared helper, and the lookup deliberately has no plaintext fallback. Accepting the stored representation as a credential would defeat hashing entirely, since anyone who could read the database could present the digest verbatim. That is ai agent identity management at the smallest useful scale: the agent's ability to act comes from a credential, the credential has exactly one accountable human, and the database does not hold a replayable copy. Enterprise ai agent identity management will eventually mean directory integration and rotation policies. It starts, though, at whether a single secret has an owner you can name.

Question Two: What Is It For

We fail this one outright, and it is the failure I would fix first if I could only fix one.

A Heym workflow has a name and a description. Neither is required to say what the workflow is for or who is responsible for it. In practice the description is empty on most workflows, including several of mine. There is no schema, no required field, no validation. Purpose is a convention at best, and a convention with no enforcement point is indistinguishable from nothing.

This is the same failure mode we documented when we audited our own instruction file for coding agents in spec driven development: a rule that cannot be violated in a single edit does not get followed. Nobody ever fails to write a purpose. They just do not write one, and no single moment is the moment they were wrong. Fixing it means making purpose a field the creation path asks for, not a paragraph in a document asking people to be thorough.

Question Three: What Is It Allowed to Touch

This is the number I expected to be uncomfortable and it was.

We have 62 registered node types. I classified each one by a single rule: can its documented operations write to, send to, or delete in a system outside Heym. Twenty-four qualify directly:

bigquery, clickhouse, codex, discord, github, googleDrive, googleSheets, grist, http, jira, linear, mcpCall, notion, opencodeGo, playwright, rabbitmq, redis, s3, sendEmail, sentry, slack, supabase, telegram, websocketSend

The indirect ones are where ai agent governance gets genuinely hard, because reach becomes transitive. Four more node types qualify indirectly, because they can reach anything the first twenty-four can: agent calls tools, execute runs a sub-workflow, plugin runs custom node code, and code runs arbitrary Python. That is 28 of 62, or 45 percent of the node catalogue, capable of an action you cannot take back. Once one agent can start another, the boundary you drew for the first one is only as narrow as the widest thing it can call, which is the delegation problem we covered in multi-agent AI systems.

Now the part that matters for ai agent permissions. Of those 28, exactly one ships closed by default: the Code node runs with codeAllowNetwork set to false, so the container has no network at all unless you deliberately turn it on. The other 27 have no equivalent switch. If a workflow contains a Slack node, that Slack node will post. That is the honest state of ai agent permissions in our product today: the boundary is whatever the workflow author drew on the canvas, and nothing narrower is applied underneath it. Isolation is a separate control from permission, and we wrote about where that line sits, badly at first, in AI agent sandbox.

Human approval exists in the product. It is a field called hitlEnabled on the agent node, and it defaults to false. So the honest sentence is: zero of our 28 outward-acting node types require human approval by default, approval is available on one node type, and you have to know to turn it on.

There is a second detail I went back and forth on including. When approval is enabled, the policy that decides which tool calls pause is written in natural language, and the executor matches it with 12 positive regular expressions such as \bask before\b and \brequires?\b.{0,24}\b(review|approval|permission)\b, against 12 negative ones such as \bnever\b.{0,24}\bask\b and \bpre[- ]approved\b. It works well in practice and it is much friendlier than writing a policy schema. It is also, structurally, prose being pattern matched, which is exactly the thing this post is complaining about elsewhere. Both facts are true at once and I would rather write that down than pretend the tension is not there.

The question people actually ask at this point is which control to reach for, and the answer is decided by one property of the action: whether you can take it back. Here is the rule we use.

The mistake is treating those four rows as one row. A team that gates everything and a team that gates nothing both end up with an agent nobody is really supervising, by opposite routes. We wrote about where approval gates belong, and where they backfire, in human in the loop AI agents. The short version is that gating everything is worse than gating nothing, because a reviewer facing forty approvals a day approves without reading and you end up with an audit trail of rubber stamps.

Question Four: How Do You Turn It Off

We pass this one, and the reason we pass it is that we got it wrong first in a related system.

Cancellation in our executor is not an in-memory flag. There is a dedicated cancellation module, a cancel bus, and a cancel_requested_at column on the active execution row, and every node checks a cancellation event between steps. That architecture exists because we learned, painfully, that anything process-local breaks the moment you run more than one worker. Our cron scheduler once replayed an entire backlog because eight uvicorn workers each kept their own idea of when the last check happened.

Two dates make the point better than the architecture description does. The cancellation module has been in the repository since the first commit on 27 April 2026, which is 120 days of it running under every workflow we execute. The audit trail, by contrast, landed this week. That gap is why I can be precise about the holes in one and confident about the other: shutdown has had four months of production to fail in and has not, and the audit trail has had four days.

The before and after is worth stating plainly, because we did not design this correctly the first time. Before: a process-local timestamp, and eight uvicorn workers each holding a different opinion about when the last check ran, which replayed an entire cron backlog. After: the schedule claims its slot in the database with a row lock, and the same handoff produces one run instead of eight. Cancellation inherited that lesson rather than learning it again.

The general shape of a working shutdown is three parts. A signal that reaches every process, which in practice means a shared store rather than a variable. A check the running work actually performs, because a signal nobody reads is decoration. And a record that the stop was requested, by whom, and when, because otherwise your incident timeline has a hole exactly where the interesting part is.

I would encourage you to actually run the test. Start a long agent, cancel it, and confirm it stopped everywhere. Shutdown procedures rot the same way backup restores do, silently, and for the same reason: nobody exercises them.

The Audit Trail Is Where Governance Becomes Checkable

Everything above is prevention. An ai agent audit trail is what you have left when prevention did not work, and it is the only part of a governance program that an incident review can actually use.

We shipped ours recently, so I can be precise about its shape and its holes. It emits one line per security-relevant action through a dedicated logger, and each line carries six things: the action name, the actor, the outcome, the target type, the target identifier, and a human-readable target name. Nothing is written to a database table. That was a deliberate call rather than an oversight, because a log line reaches the same place the access log does and does not create a second store to secure, back up and rotate.

Here is the whole record, field by field, because this is the part AI answers get wrong most often by padding it with fields nobody reads.

Two choices in it are worth stealing.

The first is aggressive redaction at the boundary. Any detail key containing password, passphrase, secret, token, api_key, private_key, authorization, cookie, credential_data or value loses its value entirely, and anything over 256 characters is truncated. Over-redaction is deliberate. An unhelpful audit line beats a credential sitting in stdout forever.

The second is what we left out. There is no client IP in our audit lines. Behind a load balancer the address is either the proxy's own or a forwarded header anyone can set, so it identifies the user without reliably identifying anything else. Logging a field that looks like evidence and is not is worse than logging nothing, because someone will eventually build a decision on it.

Now the uncomfortable number. We have 48 API router modules. Thirteen of them emit audit lines. That is 27 percent, and the thirteen were chosen because they were obviously security-relevant: authentication, credentials, teams, workflows, files, folders, data tables, vector stores, alerts, global variables, MCP, boards, and file intake. The other 35 are not all interesting, but I am certain some of them are, and I cannot tell you which without going through them one at a time.

Notable fact: Every page I read while researching this post described what an audit trail should capture. Not one of them stated what fraction of its own surfaces was instrumented. That ratio is the number that tells you whether a trail is a control or a gesture, and it is the number nobody publishes.

There is one implementation trap worth passing on, because it cost us a broken request. Our audit() function is wrapped in a try and except and never raises, which sounds like it makes call sites safe. It does not. The try block is inside the function, so it protects the body, not the arguments. A call site that passes target_name=workflow.name after the workflow has already been deleted raises an AttributeError before audit() is ever entered, and the request fails on a logging call that was supposed to be incapable of failing. Resolve your values before the call, or accept that your audit helper is a live dependency of the endpoint it observes.

For what to keep from a run beyond the audit line itself, we went through the trace and metric side in AI agent observability, and the span plumbing in OpenTelemetry for AI agents. The two are complementary: spans tell you what took nine seconds, audit lines tell you who authorised it.

Local Inference Does Not Remove the Identity Problem

I started drafting this post on the morning of 25 August 2026, and by the afternoon two hardware announcements had landed that made the section rewrite itself.

Apple announced a Mac Studio with M5 Ultra carrying up to a 36-core CPU, an 80-core GPU and up to 512GB of unified memory, described in Apple's own words as enabling users to run enormous LLMs entirely on device. It starts at 5,499 US dollars, pre-orders opened the same day, and Ars Technica read the launch as Apple leaning hard into local inference as a product category. Separately, Xiaomi showed an on-premise appliance built from three of its own XRING processors, reported as running 120-billion-parameter models entirely offline.

The privacy argument for this is real and I am not arguing with it. Prompts stay on the machine. Data does not cross a vendor boundary. IDC framed the enterprise version of the same bet as an administrator being able to spell out exactly what an agent may reach and push sensitive requests to a model on the machine.

Here is what local inference does not do. It does not shrink the agent's reach. The tools are still wired to the same APIs with the same credentials, and every one of the four questions is unchanged. The model moved. The blast radius did not.

And there is a second-order effect that I think is underrated. For a lot of teams, the outbound call to a model provider was functioning as an unofficial agent inventory. You could see agents in the API bill and in the egress log even when nobody had written them down. Move inference on-device and both signals disappear at once. The agent stops appearing in the invoice and stops appearing in the network capture, and if your register was never real, you now have no substitute for it.

So the sequencing matters. Local inference is a fine thing to adopt, and it is a bad thing to adopt before you can answer the four questions, because it removes the accidental telemetry you were relying on without replacing it.

An Agent Register You Can Actually Run

Enough argument. Here is the smallest useful version of the coverage metric, as a workflow.

It runs on a schedule, lists every workflow the owner can reach, checks each description for an owner and a purpose declaration, and reports the live ones that have neither. It needs no credentials and no model, which means it runs on a fresh instance with nothing configured.

{
  "heym": true,
  "nodes": [
    {
      "id": "reg_note",
      "type": "sticky",
      "position": {
        "x": 40,
        "y": 40
      },
      "data": {
        "label": "setupNote",
        "stickyTitle": "What this measures",
        "stickyColor": "violet",
        "stickyWidth": 320,
        "stickyHeight": 220,
        "note": "Convention: every workflow description must contain owner: and purpose:.\n\nThis run reports coverage as a percentage and lists the ACTIVE workflows that declare neither. No credentials and no model are required."
      }
    },
    {
      "id": "reg_cron",
      "type": "cron",
      "position": {
        "x": 40,
        "y": 320
      },
      "data": {
        "label": "weeklySweep",
        "cronExpression": "0 9 * * 1"
      }
    },
    {
      "id": "reg_list",
      "type": "heym",
      "position": {
        "x": 380,
        "y": 320
      },
      "data": {
        "label": "agentRegister",
        "heymOperation": "listWorkflows",
        "heymLimit": "0"
      }
    },
    {
      "id": "reg_check",
      "type": "code",
      "position": {
        "x": 720,
        "y": 320
      },
      "data": {
        "label": "RegisterCheck",
        "codeSource": "def main(params):\n    rows = params.workflows\n    declared = []\n    undeclared = []\n    for row in rows:\n        text = (row.get(\"description\") or \"\").lower()\n        has_owner = \"owner:\" in text\n        has_purpose = \"purpose:\" in text\n        entry = {\n            \"name\": row.get(\"name\"),\n            \"id\": row.get(\"id\"),\n            \"active\": row.get(\"active\"),\n            \"owner\": has_owner,\n            \"purpose\": has_purpose,\n        }\n        if has_owner and has_purpose:\n            declared.append(entry)\n        else:\n            undeclared.append(entry)\n    live_gaps = [e for e in undeclared if e[\"active\"]]\n    total = len(rows)\n    return {\n        \"total\": total,\n        \"declared_count\": len(declared),\n        \"undeclared_count\": len(undeclared),\n        \"live_gaps\": live_gaps,\n        \"live_gap_count\": len(live_gaps),\n        \"coverage_pct\": round(100.0 * len(declared) / total, 1) if total else 0.0,\n    }\n",
        "codeRequirements": "",
        "codeParameters": "{\n  \"workflows\": \"$agentRegister.workflows\"\n}",
        "codeAllowNetwork": false
      }
    },
    {
      "id": "reg_gate",
      "type": "condition",
      "position": {
        "x": 1060,
        "y": 320
      },
      "data": {
        "label": "AnyLiveGaps",
        "condition": "$RegisterCheck.result.live_gap_count >= 1"
      }
    },
    {
      "id": "reg_gaps",
      "type": "output",
      "position": {
        "x": 1400,
        "y": 180
      },
      "data": {
        "label": "GapsFound",
        "outputSchema": [
          {
            "key": "coverage_pct",
            "value": "$RegisterCheck.result.coverage_pct"
          },
          {
            "key": "live_gaps",
            "value": "$RegisterCheck.result.live_gaps"
          },
          {
            "key": "total",
            "value": "$RegisterCheck.result.total"
          }
        ]
      }
    },
    {
      "id": "reg_clean",
      "type": "output",
      "position": {
        "x": 1400,
        "y": 460
      },
      "data": {
        "label": "RegisterClean",
        "outputSchema": [
          {
            "key": "coverage_pct",
            "value": "$RegisterCheck.result.coverage_pct"
          },
          {
            "key": "total",
            "value": "$RegisterCheck.result.total"
          }
        ]
      }
    }
  ],
  "edges": [
    {
      "id": "reg_e1",
      "source": "reg_cron",
      "target": "reg_list"
    },
    {
      "id": "reg_e2",
      "source": "reg_list",
      "target": "reg_check"
    },
    {
      "id": "reg_e3",
      "source": "reg_check",
      "target": "reg_gate"
    },
    {
      "id": "reg_e4",
      "source": "reg_gate",
      "target": "reg_gaps",
      "sourceHandle": "true"
    },
    {
      "id": "reg_e5",
      "source": "reg_gate",
      "target": "reg_clean",
      "sourceHandle": "false"
    }
  ]
}

Three things in that file are the argument in miniature. codeAllowNetwork is false, so the step doing the checking cannot phone anywhere. The check runs on a cron rather than on someone remembering, which is the difference between a metric and an intention. And the output is a percentage rather than a list, because a list of forty workflow names gets skimmed and a coverage number of 18 percent does not.

I ran the exact code above, extracted from the file rather than retyped, against three input sets before publishing. Five workflows returns coverage_pct: 40.0 with two live gaps. Thirty workflows returns 26.7 percent with fifteen. An empty instance returns 0.0 rather than a division error, which is the case worth testing because it is the one you hit on day one. Swap the owner: and purpose: convention for whatever your team already uses. The convention is not the point. Having one that is measured weekly is the point.

This is also a nice illustration of ai agent lifecycle management done cheaply. You do not need a registry product to know your coverage. You need a scheduled job and a convention. The rest of ai agent lifecycle management, rotation, decommissioning, re-approval after an edit, is the same shape: a convention plus something that measures it on a timer. One caution before you point an agent at your own register: anything that reads workflow descriptions is reading text a person wrote, and text a person wrote is an injection surface, which is why the checking step here is deterministic Python rather than a model. We went through why that distinction matters in prompt injection.

What to Do This Week

The order matters more than the completeness. Each of these is a couple of hours, not a quarter.

1. Answer the four questions for one agent. The one that touches your most sensitive system. Write the answers down. Twenty minutes, and it will tell you which question your organisation cannot answer at all.
2. Make purpose a required field. Not a suggestion in a wiki. Somewhere the creation path asks for it, so a blank one is visible.
3. Enumerate what leaves your system. Mark each integration read, write, or destructive. Our number was 28 of 62, and nobody on the team had guessed it correctly beforehand.
4. Put approval only in front of the destructive ones. Write, delete, send outward. Never reads, never arithmetic. See LLM guardrails for how the action layer fits with the input and output layers.
5. Emit one audit line per security-relevant action, then count your coverage as a fraction of your surfaces. Publishing that fraction internally does more than adding another field.
6. Test the shutdown on a real run. Cancel a live agent and confirm it stopped in every process.

If you run the register template above on a schedule, step three and step five both get a number attached to them, and numbers are what turn this from a policy conversation into an engineering backlog. The natural next move is to alert on the number rather than read it, which is the same discipline we applied to spend in LLM cost alerts: a metric nobody is paged about is a metric nobody acts on, and that is true of ai agent governance coverage exactly as it is true of a monthly bill.

What This Post Is Not

Three exclusions, stated so you know what you are not getting.

It is not regulatory advice. The EU AI Act implementation timeline has milestones that apply to some deployments and not others, and whether yours is in scope is a question for someone qualified to answer it.

It is not a vendor evaluation. If your job this quarter is choosing a runtime enforcement product, the enforceability ladder in the ARMO piece linked earlier is a better instrument than anything here.

And it is not a claim that our platform is well governed. Two of the four questions came back Pass, one came back Partial, and one came back Fail. I published the numbers because the numbers are the contribution, not because they are flattering. The Cloud Security Alliance's work on non-human identity and agentic AI governance calls the general condition a governance vacuum, and I think that is fair, including for us.

The thing I actually believe, and the reason I wanted to write this down: choosing a model is the easy part. Keeping an agent inside its boundary over time, as the team changes and the workflow gets edited by someone who was not there when it was designed, is the hard part. Identity and access management teams will end up owning agent inventories, credential rotation and decommissioning procedures, because those are the same problems they already solved for service accounts. The sooner the engineering side hands them a register that is actually populated, the less painful that handover will be.

Frequently Asked Questions

What is AI agent governance?

AI agent governance is the set of records and controls that make an autonomous agent accountable: a named human owner, a written purpose, an enforced permission boundary, and a shutdown procedure someone can execute. The distinction that matters in practice is between governance you can state and governance you can check. A policy saying agents must not delete production data is a sentence. A permission boundary that refuses the delete call is a control. Both belong in a governance program, but only the second one produces evidence after an incident.

How is agent governance different from AI governance?

Traditional AI governance was written for models that produce text. It covers acceptable use, content filtering, bias review, and data handling. Agents break that frame because they take actions in other systems. A model that writes a bad paragraph is a quality problem. An agent that sends the paragraph to two thousand customers is an incident. So agent governance adds the three things model governance never needed: an identity for the actor, a boundary on what it may touch, and a record of what it did.

What should an AI agent audit trail capture?

Six fields cover most of what an investigation needs: the action name, the actor identity, the outcome, the target type, the target identifier, and a human-readable target name. Everything else is optional. What must never appear is the payload of the thing being protected. Our own audit helper redacts any field whose key contains password, secret, token, api_key, credential_data, or value, and truncates anything over 256 characters, because an unhelpful log line is better than a leaked credential in stdout.

How do you shut down an AI agent?

You need three things and most teams have one. First, a stop signal that reaches every process running the agent, which in a multi-worker deployment means a shared store rather than an in-memory flag. Second, a check the running work actually performs, because a signal nobody reads is decoration. Third, a record that the stop was requested, by whom, and when. Our executor stores a cancel_requested_at timestamp on the active execution row and every node checks a cancellation event between steps.

Does running the model locally solve the governance problem?

No, and it quietly makes one part harder. Local inference removes the prompt from the network, which is a real privacy gain. It does not shrink what the agent can reach, because the tools are still wired to the same APIs and the same credentials. It also removes the outbound call to a model provider that many teams were unknowingly using as their agent inventory. When Apple ships a desktop with 512GB of unified memory for on-device models, the agent stops appearing in the bill and in the egress log at the same time.

What is AI agent sprawl?

AI agent sprawl is the state where an organization has more agents than it has records about agents. It is usually measured as a count, which is the wrong unit. The useful measurement is coverage: what percentage of your running agents have a named owner and a written purpose. A team with forty agents and forty owner records is in better shape than a team with twelve agents and no register, even though the second number sounds tidier.

Written by Ceren Kaya Akgün, who works on the Heym execution engine. Heym is our own product, so treat the platform sections as interested rather than neutral. Every count in this post was taken on 25 August 2026 against our repository at commit c2a8b6b7, the classification rule for the 28 node types is stated in full so you can disagree with it, and the template Python was executed before publication rather than written to look plausible.