Spec Driven Development: What Our Agents Ignored

TL;DR: Spec driven development promises that a written specification governs what your coding agents build. I audited ours against the repository it actually governs. The file is 203 lines and has been read by coding agents across 1,133 commits in 117 days. Three rules were followed perfectly. One rule was broken in 85 of 272 files, including inside the very directory that one of the obeyed rules created. The difference between them was not length, age or importance. It was whether a single edit could break the rule.

Key Takeaways:

• Rules that name a file, a destination and a forbidden pattern held at 100% across 1,133 commits, one of them for 53 days and one for only 10
• A rule stating a global property, that Vue files stay under 300 lines, has been in the spec since the first commit and is violated in 31% of files
• The strongest evidence is the control case: 18 of the 59 components created by the obeyed rule break the ignored one
• The popular claim that agents stop reading past roughly 150 lines does not survive contact with our file, which is 203 lines and grew 51% while its precise rules kept holding
• If a rule cannot be violated in one edit, no amount of rewording will save it, and it needs a tool or a scheduled check instead

Table of Contents

• The Question at the Top of the Search Results
• What Spec Driven Development Actually Means
• The Spec I Tested
• Three Rules That Held
• The Rule That Died
• The Control That Settled It
• Length Was Not the Variable
• The Shape of a Rule That Survives
• Spec Driven Development Best Practices From the Audit
• Where a Tool Beats a Sentence
• Spec Driven Development vs Vibe Coding
• Giving a Rule a Moment
• Where This Does Not Apply
• What I Would Tell Birgitta Böckeler
• FAQ

The Question at the Top of the Search Results

The best article I found on spec driven development ends by admitting it cannot answer the question I actually had. Birgitta Böckeler, writing on martinfowler.com in October 2025, compares three of the major tools in this space and then says plainly: "Until I hear usage reports from people using them for a period of time on a 'real' codebase, I still have a lot of open questions about how this works in real life."

Ten months later, that request is still open. I read through the pages that rank for this topic and they are almost all definitions or tool comparisons. IBM explains the concept well. Microsoft explains why it matters. GitHub ships a toolkit. None of them tell you what fraction of a real specification a real coding agent actually follows over a real quarter, because nobody has published that number.

We were in an unusual position to answer it. Our product is built by coding agents working against a specification file that has been in the repository since the first commit. So I stopped reading and went to count.

This post is for engineering leads who already have agents committing to their codebase and want to know whether the instruction file they wrote is doing anything. If you are looking for a comparison of Kiro against Spec Kit against Tessl, Böckeler already wrote that one and it is better than anything I would produce.

What Spec Driven Development Actually Means

Spec driven development is a way of building software where a written specification is agreed first and treated as the source of truth, and the code becomes the artifact generated from it or verified against it rather than the record of what was decided.

It inverts the usual order. Instead of writing code and describing it afterwards, you write a specification that both humans and agents treat as the source of truth, and the code becomes the thing that gets generated against it or checked against it. IBM defines it as a methodology where a detailed specification is authored and agreed upon before development begins, serving as a single source of truth for what to build and how to build it.

GitHub's Spec Kit puts it more ambitiously, describing a spec that becomes the contract your tools and agents generate from, and Microsoft's engineering blog frames the same idea as aligning humans and AI on one source of truth before prompting rather than after.

There are recognised levels of rigour. A January 2026 paper by Deepak Babu Piskala sets out three: spec-first, where the spec provides initial clarity and is then allowed to age; spec-anchored, where the spec evolves with the software and automated tests keep the two in sync; and spec-as-source, where changing the spec is the only way to change the code. Most teams doing spec driven development with ai agents in practice are somewhere in the first two.

None of this is new in spirit. IBM traces the lineage directly to test-driven development, where Kent Beck asked teams to state the expected outcome first, and to behaviour-driven development, which asked them to state the behaviour first. What changed in 2026 is who reads the answer. Red Hat's developer team made the same point in October 2025, describing the human as a conductor laying out a score for an ensemble that will otherwise improvise. For twenty years the audience for a specification was a person who could ask a follow-up question. Now it is a process that cannot.

That taxonomy is useful and I am not going to relitigate it. My question sits underneath it. All three levels assume the agent reads the specification and acts on it. That assumption is the one nobody has measured, and it is the one that decides whether any of this works.

The Spec I Tested

Our specification is a file called AGENTS.md in the root of the repository for Heym, the workflow automation product I work on. Coding agents read it before they touch anything. Here is what it is, measured on 21 August 2026:

Three things make this a fair test rather than a flattering one.

The counting itself is four commands, which is worth stating because the whole point is that you can run it on your own repository this afternoon:

$ wc -l AGENTS.md
203 AGENTS.md
 
$ find frontend/src -name "*.vue" | wc -l
272
 
$ find frontend/src -name "*.vue" -exec wc -l {} + | awk '$2!="total" && $1>300' | wc -l
85
 
$ grep -c "selectedNode.type" frontend/src/components/Panels/PropertiesPanel.vue
0

For each rule I picked the one grep or count that would expose a violation, ran it across the whole tree rather than a sample, and recorded the result before deciding what it meant. Rule introduction dates come from git log --follow -- AGENTS.md, checking each of the 17 revisions for the presence of the rule text.

The file has real authority. It is loaded as a hard directive at the start of every session, not offered as background reading. If an instruction file was ever going to be followed, this is the setup you would want.

The rules are heterogeneous. Some name a file. Some state a property. Some are about formatting, some about architecture, some about what you must never commit. That variety is what makes the comparison possible, because the agents, the repository and the time window are identical across all of them.

And I did not choose the sample after looking at the results. I took the rules that were mechanically checkable, checked all of them, and wrote down what came back including the parts that are embarrassing.

Three Rules That Held

Start with the good news, because it genuinely surprised me.

Rule one. The specification says that PropertiesPanel.vue must stay a thin shell, that node configuration UI belongs under propertiesPanel/nodes/ with one component per node type, and that you must not add selectedNode.type branches to the panel. This rule entered the file on 29 June 2026, so it has had 53 days and several hundred commits to erode.

It did not erode. PropertiesPanel.vue is 43 lines long. It contains zero selectedNode.type branches. The destination directory holds 59 components, exactly as instructed.

Rule two. The alerts evaluator owns the state machine and nothing else, and each alert type must live in its own module under services/alerts/types/ with a registry entry. This rule is much younger, added on 11 August 2026, so it has had about ten days.

The evaluator is 351 lines with zero alert_type == branches, and there are 6 type modules in the folder.

Rule three. The workflow executor keeps orchestration, retries, tracing and cancellation, and node specific execution logic goes into a handler module per node type. There are 62 handler modules. The executor still contains 7 comparisons on node type, and I checked each one: they are branch skipping for conditions and switches, error handler routing, and two short circuits for output nodes. Every one of them is orchestration, which is the executor's job under the same rule. Compliant as written.

Three rules, three clean results, no tooling behind any of them. Nothing in our linting, our type checking or our test suite would fail if an agent broke any of these. They held on prose alone.

The Rule That Died

Now the part I did not enjoy writing.

Buried in a bullet list of TypeScript conventions is this: Vue: Composition API with <script setup>, file names: PascalCase, max 300 lines.

That rule has been in the specification since the first commit on 27 April 2026. It is present in all 17 revisions. It has had the entire life of the repository, every agent, every session, every review.

85 of our 272 Vue files exceed 300 lines. That is 31%.

The largest is DebugPanel.vue at 4,516 lines, just over fifteen times the limit. Behind it, CredentialDialog.vue at 3,820 and ExpressionInput.vue at 3,143.

It is not a historical problem that we have since outgrown. Of the 131 Vue files created since 21 June 2026, 26 were already over 300 lines on the day they were committed. One in five new files is born in violation of a rule that every agent read that morning.

And there is no safety net. I checked our ESLint configuration for a max-lines rule and the count of occurrences is zero. Nothing in the toolchain has ever mentioned this.

The Control That Settled It

Here is the finding that turned this from an anecdote into something I trust.

The obeyed rule created a directory. Fifty nine node components now live in propertiesPanel/nodes/, and they exist precisely because the agents followed the instruction to put them there.

Eighteen of those 59 components are over 300 lines. The largest, GithubNodeProperties.vue, is 1,116 lines. The median across the folder is a healthy 165, so this is not a folder full of monsters, but nearly a third of it breaks the other rule.

Sit with that for a second. Same repository. Same specification file. Same agents, in the same sessions, writing the same files. The agents built exactly the structure one rule asked for, and filled it with files that violate another rule from the same document, sometimes in the same commit.

Whatever explains this, it cannot be that the agents did not read the file, and it cannot be that they lacked capability or context. They followed one sentence and ignored another sentence four screens above it.

Length Was Not the Variable

There is a claim circulating that explains all of this away, and I wanted it to be true because it would have been an easy fix. A developer writing in July 2026 put it crisply: "Then we learned that if one of those documents exceeds 150 lines, the agent will ignore most of it."

Our file is 203 lines. It grew 51% over four months, from 134 lines to 203. If the claim held, compliance should have degraded as the document grew, and the rules furthest down the file should be the weakest.

Neither is true here. The alerts rule was added when the file was already 175 lines, which is well past the supposed cliff, and it holds at 100%. The rule that failed is not at the bottom of a bloated document. It has been near the top since the file was 134 lines, back when the document was comfortably under the limit and there was almost nothing competing for attention.

I am not claiming there is no length at which an instruction file stops working. I am claiming that at 203 lines, length did not predict a single thing about which rules survived, and that a team debugging spec drift by deleting paragraphs is optimising the wrong variable.

The Shape of a Rule That Survives

Line the rules up next to each other and the pattern is not subtle.

The rules that held share a structure. Each one names the file where the violation would happen, names where the code should go instead, and names the exact pattern that is forbidden. Each one can be evaluated in the moment an agent is editing that file, using only what is on screen.

The rule that failed states a property of a finished artifact. And here is the mechanical reason it cannot work: no single edit violates it. You open a 290 line component, add fifteen lines of perfectly reasonable markup, and at no point in that edit did you do something wrong. The file is now 305 lines. Nobody broke a rule. The rule is broken.

I have started calling this the single-edit test, and it is the only screening question we now apply to a proposed rule: can one edit break this rule, and would the agent know in that edit? Two yeses and the rule can live in prose. Anything else needs a machine.

That is what people are describing when they say specs drift. The word makes it sound like slow erosion of intent, and I think it is something more specific than that. Drift is what a rule does when it has no moment of failure. It is not that the agent forgot. It is that there was never an instant where the agent was doing the wrong thing and could have known.

The frustration is widely shared and usually blamed on the model. The most upvoted post I found on the subject, with 194 votes on r/programming in February 2026, argues that the method fails when the author is too confused to write the spec. Our data points somewhere else. The rule that failed was not confused. It was perfectly clear and completely unenforceable.

A commenter on r/ClaudeCode got to this from the other direction in February 2026: "SDD only works if specs are living + enforced, not static docs. You still need tests/CI to handle LLM randomness, and drift is inevitable without any guardrails." I would refine that slightly based on what I measured. Enforcement does not always mean CI. Three of our rules are enforced by nothing but their own wording, and they are at 100%. What they have is not a guardrail, it is a trigger.

Rewriting the rule that died

We are keeping the 300 line target, because it is a good target. But it stops being a spec rule.

The old wording, inside a bullet list of conventions:

Vue: Composition API with <script setup>, file names: PascalCase, max 300 lines

The replacement is two separate things. A rule with a moment:

When a component passes 300 lines, extract the next section into a
sibling under the same folder before adding to it. Do not grow
PropertiesPanel node components past 300 lines: split the operation
specific form into its own component and import it.

And an ESLint max-lines warning, so that the aggregate property gets checked by the only thing that can see the aggregate. The sentence handles the edit. The tool handles the file.

Spec Driven Development Best Practices From the Audit

Six spec driven development best practices came out of this measurement, and every one of them is a change we actually made rather than something I think sounds sensible.

Write rules that can be broken in one edit. This is the whole finding compressed. Before adding a rule, ask what the agent would be doing at the exact second it violated it. If you cannot describe that second, the rule will not survive.

Name the file. Every rule that held names a specific path. Every rule that failed applies to a category. Specificity is not pedantry here, it is what makes a rule addressable at all.

Say where the code should go instead. A prohibition without a destination leaves the agent to invent one, and the invented one differs each time. Our three surviving rules all pair the forbidden pattern with the folder and the registry entry.

Give aggregate rules a tool. Anything about size, coverage, duplication or naming across many files cannot be a sentence. It needs a linter, a formatter or a scheduled check.

Measure before rewriting. I would have guessed our compliance was around 80% across the board. It is 100% in one shape of rule and 69% in another. Guessing would have sent me to reword the wrong things.

Delete rules you will not enforce. A rule at 31% violation is teaching every reader that the document is advisory. That lesson spreads to rules you care about.

For agents.md best practices specifically, one more thing surfaced that I did not expect. The rules we wrote most recently are the most precise ones, because by August we had learned what a rule needs to look like. The same question applies to every other artifact an agent reads before it works, including agent skills, which are instruction files by another name and inherit exactly the same problem. Our oldest rules are our vaguest. If you inherited an instruction file, audit it oldest first.

Where a Tool Beats a Sentence

The Python side of our codebase makes the opposite case, and it is worth being honest about how it wins.

Our specification says the Python line length is 100 columns. Compliance is total: ruff format --check . reports 623 files already formatted and ruff check . passes clean. Perfect adherence to a global property, which is exactly what I said prose cannot do.

Except look at how it is achieved. The rule E501, which is the check that would actually warn you about a long line, is in our ignore list. Nobody is ever told they wrote a long line. The formatter simply rewrites the code on the way past.

That is not compliance. That is the absence of an opportunity to fail. And it is the correct answer for this class of rule, because a formatter can see the whole file while a person or an agent is looking at one hunk. When you cannot give a rule a moment, give it a machine, and stop pretending the sentence is doing the work.

Spec Driven Development vs Vibe Coding

The framing of spec driven development vs vibe coding is usually about discipline: one is careful and one is not. Our measurement suggests something less flattering to both sides.

The DeepLearning.AI course on the topic opens by saying vibe coding is fast but often produces code that does not match what you asked for, and that writing the spec first is the disciplined alternative. That is fair, and if you want the fuller picture of the thing being contrasted, we wrote about vibe coding separately.

What the audit adds is this. Writing the specification is the easy half, and it is the half that feels like progress. Our file grew 51% in four months. Every one of those additions felt like governance at the time. But 31% of one rule's surface area was quietly failing that entire time, and nobody noticed, because a document that is not followed reads exactly like a document that is.

Vibe coding at least tells you honestly that nothing is enforcing anything. An unenforced spec tells you the opposite while doing the same amount of work. That is the real answer to people asking about spec driven development disadvantages: the method is not fragile, but the confidence it creates is, and confidence without measurement is the failure mode that costs the most.

Giving a Rule a Moment

The fix for aggregate rules is to create the moment artificially. Run the check on a schedule and route the result to a person.

We build these as workflows, which is convenient because it is our own product, but the shape is portable to anything that can run a script on a timer. The version below is deliberately small: a step that holds a file inventory, a sandboxed Python step that applies one rule, a condition, and a review gate before anything is filed. The spec driven development workflow you actually want is probably this plus your own rules, not something more elaborate.

The Python in this template ran before publication. Against the sample inventory it reports 2 violations out of 3 files. Against our real inventory of 272 Vue files it returns 85 violations at 31.2%, with DebugPanel.vue at 4,516 lines as the worst, which is where every number in this article came from.

{
  "heym": true,
  "nodes": [
    {
      "id": "spec_note",
      "type": "sticky",
      "position": {
        "x": 40,
        "y": 40
      },
      "data": {
        "label": "setupNote",
        "stickyTitle": "What this check is for",
        "stickyColor": "violet",
        "stickyWidth": 320,
        "stickyHeight": 240,
        "note": "A rule that lives only in a markdown file has no moment of failure, so it drifts.\n\nThis workflow gives one such rule a moment. CheckSpecRules runs with Allow network off. DraftSpecReport has human review on, so nothing is filed without a person reading it first."
      }
    },
    {
      "id": "spec_input",
      "type": "textInput",
      "position": {
        "x": 40,
        "y": 330
      },
      "data": {
        "label": "FileInventory",
        "text": "[{\"path\":\"src/components/Panels/DebugPanel.vue\",\"lines\":4516},{\"path\":\"src/components/Credentials/CredentialDialog.vue\",\"lines\":3820},{\"path\":\"src/components/ui/Button.vue\",\"lines\":58}]"
      }
    },
    {
      "id": "spec_check",
      "type": "code",
      "position": {
        "x": 400,
        "y": 330
      },
      "data": {
        "label": "CheckSpecRules",
        "codeSource": "import json\n\n\ndef main(params):\n    limit = int(params.limit)\n    files = json.loads(params.inventory)\n    over = [f for f in files if f[\"lines\"] > limit]\n    over.sort(key=lambda f: -f[\"lines\"])\n    return {\n        \"checked\": len(files),\n        \"violations\": len(over),\n        \"percent\": round(100.0 * len(over) / len(files), 1) if files else 0.0,\n        \"worst\": over[0] if over else None,\n        \"top\": [{\"name\": f[\"path\"].split(\"/\")[-1], \"lines\": f[\"lines\"]} for f in over[:5]],\n    }\n",
        "codeRequirements": "",
        "codeParameters": "{\n  \"inventory\": \"$FileInventory.text\",\n  \"limit\": \"300\"\n}",
        "codeAllowNetwork": false
      }
    },
    {
      "id": "spec_gate",
      "type": "condition",
      "position": {
        "x": 760,
        "y": 330
      },
      "data": {
        "label": "AnythingBrokenToday",
        "condition": "$CheckSpecRules.result.violations >= 1"
      }
    },
    {
      "id": "spec_agent",
      "type": "agent",
      "position": {
        "x": 1120,
        "y": 230
      },
      "data": {
        "label": "DraftSpecReport",
        "hitlEnabled": true,
        "systemPrompt": "You report on one rule from the team specification. Name the rule, give the count and the percentage, list the worst offenders, and propose one concrete change to either the code or the wording of the rule. Do not moralise and do not pad.",
        "userPrompt": "Rule: Vue files stay under 300 lines.\nResult: $CheckSpecRules.result"
      }
    },
    {
      "id": "spec_clear",
      "type": "output",
      "position": {
        "x": 1120,
        "y": 430
      },
      "data": {
        "label": "NothingToReport",
        "outputText": "Rule held. $CheckSpecRules.result.checked files checked, zero over the limit."
      }
    }
  ],
  "edges": [
    {
      "id": "spec_e1",
      "source": "spec_input",
      "target": "spec_check"
    },
    {
      "id": "spec_e2",
      "source": "spec_check",
      "target": "spec_gate"
    },
    {
      "id": "spec_e3",
      "source": "spec_gate",
      "target": "spec_agent",
      "sourceHandle": "true"
    },
    {
      "id": "spec_e4",
      "source": "spec_gate",
      "target": "spec_clear",
      "sourceHandle": "false"
    }
  ]
}

Two design decisions in there are worth stating. The code step runs with network access off, because a rule checker has no reason to call anything, and a step that cannot reach the network cannot leak your inventory. The reporting step has human review enabled, so a person sees the result before it becomes a ticket. That matters more than it sounds: the useful output of this check is often "the rule is wrong" rather than "the code is wrong", and only a person can make that call. We wrote about where to put those approval gates in more detail.

If you already run agents against your codebase in the background, this slots in next to them. Our own setup has background coding agents and Codex running as a workflow step doing the work and an adversarial code review pass checking it, and this compliance check is the third leg: not is this change correct, but is this change consistent with what we said we were building.

Where This Does Not Apply

Some honest limits, because a measurement with no boundaries is marketing.

This is one repository. It is 1,133 commits, three agent tools and four months, which is enough to see a pattern and not enough to call it a law. If your team runs a different toolchain, your numbers will differ.

It only covers mechanically checkable rules. Our specification also contains judgement calls about API design and error handling that I have no way to score, and those may behave completely differently.

It says nothing about whether the specification made the software better. I measured adherence, not outcomes. It is entirely possible to have 100% compliance with rules that were a bad idea, and Böckeler's warning about a false sense of control applies to precise rules more than vague ones, because precise rules are the ones you stop questioning.

And this is not an argument for more rules. Our finding is that a small number of well shaped rules outperform a larger number of well intentioned ones. If your instruction file is growing every week, the audit that matters is not what to add.

What I Would Tell Birgitta Böckeler

She asked for usage reports from people running this on a real codebase for a period of time. Here is ours, compressed to one sentence: the specification worked, and the parts of it that failed were the parts that could not have worked, no matter who wrote them or how clearly.

That is a more encouraging answer than I expected to be writing. Going in, I thought I would find that agents skim, that instruction files are theatre, and that the honest recommendation was to give up on prose and put everything in CI. Three rules at 100% compliance across 1,133 commits, two of them added recently enough that I could watch them take effect, says otherwise. Written rules do govern agent behaviour. They just have to be the right shape.

I also think the shape is going to keep narrowing, and our own file shows the direction. Our April rules are the vague ones. Our June and August rules name a file, a folder and a forbidden pattern, because by then we had learned what survives. If that pattern holds across teams, instruction files will get shorter and more mechanical rather than longer, and the size and coverage rules will migrate out of prose into linters where they belonged all along. The interesting question for 2027 is not how to write a better specification. It is which half of today's specification stops being prose at all.

If you take one thing from this: go and count. Pick the three rules in your own file that you would swear everyone follows, run one grep each, and see which of the three is decorative. It took me an afternoon and it changed how we write rules.

FAQ

What is spec driven development? Spec driven development is a way of working where a written specification, not the code, is the thing everyone agrees on first. The code is then generated or verified against it. IBM describes it as authoring a detailed specification that is agreed upon before development begins, so it becomes the single source of truth for what to build. The reason it came back into fashion in 2026 is that coding agents write a lot of code very quickly, and a team with no agreed description of the target ends up reviewing output it never specified.

Do AI coding agents actually follow the spec? Some rules yes, some rules no, and the difference is not random. We audited a 203 line specification across 1,133 commits of our own product. Three rules that named a specific file, a specific destination and a specific forbidden pattern were followed in every case. One rule that stated a global property, that Vue files stay under 300 lines, was broken in 85 of 272 files. Same document, same agents, same commits.

Why do specs drift over time? Because most rules have no moment of failure. A rule like keep files small is never violated by any single edit. You add nine lines to a file that is already long, and nothing in that moment is wrong. The violation only exists in aggregate, and an agent working one change at a time cannot see aggregate. Rules that can be broken in a single edit, like do not add a branch to this named file, get caught by the agent in the moment it is about to break them.

Does the length of an AGENTS.md file matter? Less than people claim. There is a popular idea that once an instruction file passes roughly 150 lines the agent starts ignoring most of it. Our file is 203 lines and grew 51% over four months, and its most specific rules held at 100% compliance the entire time. Length was not the variable that predicted whether a rule survived. Shape was. A short vague rule fails and a long precise one holds.

What makes a good spec rule for AI agents? Name the file, name the destination, and name the forbidden pattern. Do not add alert_type branches to the evaluator, put each type in services/alerts/types and register it, is a rule an agent can evaluate at the exact moment it is editing that file. Keep it under 300 lines is not, because nothing tells the agent when it crossed the line. If a rule cannot be violated in one edit, it needs a tool rather than a sentence.

What are the disadvantages of spec driven development? The biggest one is false confidence. A specification that nobody enforces reads exactly like one that everybody follows, so teams believe they have governance when they have a document. There is also real maintenance cost, and the honest answer from practitioners is that a spec you do not keep updated becomes actively misleading. The method is worth it when rules are specific and enforcement exists somewhere, and it is theatre when neither is true.

How do I audit my own specification? Go through the instruction file your agents read and mark every rule as either checkable in a single edit or only checkable in aggregate. Then count the current violations of the aggregate rules. In our case that took an afternoon and produced two numbers that changed how we write rules: 100% compliance for rules naming a file and a forbidden pattern, and 85 violations out of 272 files for a rule stating a global property. The counting matters more than the rewriting, because it tells you which rules are already decorative.

What Is Heym?

Heym is a self-hosted AI workflow automation platform. You build workflows on a canvas out of triggers, language model steps, sandboxed code steps, conditions and human approval gates, and run them on your own infrastructure. The compliance check in this article is a Heym workflow, and so are the background coding agents and the review passes that work on the repository it audits.

If you want to run the same audit on your own specification, start with the template above and replace the inventory step with your own file list. It will take longer to read the result than to build it.