What are LLM cost alerts?

LLM cost alerts are threshold rules that watch how much you spend on language model calls and tell you when the spend crosses a line you set. The useful version measures spend over a time window rather than per call, because a single expensive call is normal and a hundred of them inside ten minutes is a problem. A cost alert needs three things to be worth having: a token or dollar figure, a window to measure it over, and a rule for what happens after it fires so it does not repeat every check.

Why do static LLM cost thresholds keep misfiring?

Because the metric is non-stationary. CPU on a fixed server sits in a stable band, so a threshold set in March still means something in September. LLM spend does not behave that way: swapping a model changes the per-token price outright, editing a prompt changes token volume, adding a tool call multiplies the number of round trips per run, and traffic growth scales all of it. A dollar threshold that was generous in March can be crossed by a routine model upgrade with no incident behind it. The fix is not a cleverer number, it is deciding what the threshold is protecting against and measuring it over a window that matches how fast the thing you fear actually moves.

What time window should an LLM cost alert use?

Match the window to the failure you are trying to catch. A runaway loop burns budget in minutes, so a 10 to 15 minute window catches it while there is still something to save. A gradual regression from a prompt change shows up over hours, so a 24 hour window suits a daily budget ceiling better. A window that is too short fires on ordinary bursts. A window that is too long means the money is already gone by the time anyone hears about it. Setting one alert of each kind is usually better than trying to make a single window do both jobs.

Should an alert fire once or keep notifying?

Fire once, then stay quiet until the metric recovers, as the default. An alert checked every 60 seconds on a genuinely broken workflow produces 60 notifications an hour, and the predictable result is that somebody mutes the channel. Once the condition clears, the alert resets and becomes eligible to fire again, which means a second incident is still audible. Repeating on a fixed interval is worth having as an option for conditions nobody can act on immediately, but it should be a deliberate choice rather than the default behavior.

What should an AI workflow alert on besides cost?

Four metrics cover most of what actually goes wrong. Error count over a window catches a burst of failures without paging on every single one. Run duration catches the slowdown that follows a model swap or a retry storm, and is best measured at p95 or as the slowest run rather than the average, which hides outliers. Token or dollar spend catches budget problems. Execution count catches the trigger that has started firing far more often than it should, which is frequently the cause behind the other three rather than a separate problem.

How do you avoid alert fatigue when setting thresholds?

Test the threshold against history before you save it. Most alerting is configured by guessing a number and then discovering over the following week whether it was right, which is an expensive way to learn. Replaying a proposed condition against real past runs answers the question immediately: a rule that would have fired four hundred times in the last day is noise, and you find that out before it wakes anyone rather than after. Beyond that, fire once until recovery, require a minimum sample count on duration alerts so a quiet window cannot fire on a single slow run, and attach enough context to each firing that the first question after being paged is already answered.

LLM Cost Alerts: Why Static Thresholds Break

Most teams building AI workflows learn about their spend the same way: a bill arrives, or somebody opens a dashboard on a Monday and notices the line went vertical on Thursday.

The obvious fix is an alert. Set a dollar figure, get told when you cross it. Then the alert fires on a Tuesday afternoon when nothing is wrong, gets muted, and stops being an alert.

This is not a discipline problem. It is a property of the metric. Most LLM cost alerts are built the way server alerts are built, and the number they watch does not behave the way a server metric behaves.

Why a static LLM spend threshold breaks

Classic infrastructure alerting works because the metrics are stationary. CPU on a fixed instance sits in a band. Disk fills at a rate you can extrapolate. A threshold you set in March still means roughly the same thing in September, because the thing generating the number has not changed shape.

LLM spend has none of that stability. Four things move it independently:

Model choice. Switching from one model to another can change the per-token price by an order of magnitude, in either direction, with no change to what the workflow does.
Prompt length. Adding context, a longer system prompt, or a retrieved document changes token volume per call without touching the number of calls.
Tool calls per run. An agent that used to answer in one round trip and now makes four has quadrupled its cost per run while the run count stays flat.
Traffic. Everything above multiplied by however many times the workflow ran.

A dollar ceiling set against last month's behavior can be crossed by a routine model upgrade with no incident behind it at all. The alert was not wrong. It was measuring a number whose meaning had changed underneath it. This is why an LLM spend threshold copied from CPU monitoring practice tends to misfire within weeks.

The answer is not a cleverer threshold. It is being specific about which failure you are trying to catch, and picking a window that matches how fast that failure moves.

Four things worth alerting on

Cost is the one people ask about first, but on its own it is a lagging signal. By the time spend is visibly wrong, the cause has usually been running for a while. Useful AI workflow monitoring covers three other metrics that catch the same incidents earlier or explain them afterwards. A workflow error rate alert, in particular, will often fire before the cost one does.

What each metric catches, and how fast

Execution count

A trigger firing too often

Usually the cause behind the other three. A workflow that ran 2000 times instead of 20 explains the cost spike and the error burst at once

Error count

A burst of failures in a window

One failed run is noise. Twelve inside ten minutes is an incident. Counting over a window is what separates the two

Run duration

The slowdown after a change

Retry storms and model swaps show up here first. Measure the slowest run or p95, not the average, which hides the tail

Token or dollar spend

The budget itself

Lagging but unambiguous. Best as a ceiling over a long window rather than a tripwire over a short one

The approach this replaces

Per-run alerting

Notify on every failure

What most platforms give you by default. It pages on transient errors and goes silent on the gradual regressions that cost the most

The pattern across all four is the same: measure over a window, not per event. A single failed run, a single slow run, and a single expensive call are all normal. Their frequency inside a bounded period is the signal.

Choosing the window

The window is the part people skip, and it does more work than the threshold number. It is the single biggest difference between alert thresholds for AI workflows that hold up and ones that get muted.

Match it to how fast the failure you fear actually moves:

A runaway loop burns budget in minutes. A 10 to 15 minute window catches it while there is still something left to save.
A prompt regression leaks money slowly. A 24 hour window works better as a daily ceiling.
A dependency outage produces errors in a burst. Ten to thirty minutes is usually enough to separate a real outage from three unlucky retries.

Too short and you fire on ordinary variance. Too long and the money is already spent by the time anyone hears about it.

When one window cannot serve both purposes, use two alerts. A tight window on execution count to catch the loop, and a wide window on spend as the backstop, cost less in attention than one badly compromised rule.

The aggregate matters more than people expect

For duration alerts specifically, the aggregate you choose changes what you are measuring.

Average hides exactly what you want to see. Ninety fast runs and ten timeouts average out to something unremarkable.

Maximum shows the worst case, which is useful, but in a quiet window the maximum is just the one run that happened. Fire on that and you are alerting on a sample size of one.

p95 is usually the right default for anything with real traffic. It tracks the tail without letting a single outlier drive the decision.

Whichever you pick, set a minimum number of runs before the alert is allowed to fire. Without it, the first slow run in an otherwise empty overnight window trips a threshold that means nothing.

Fire once, then be quiet

An alert that has fired has done its job. Repeating it until somebody acts is a design that reliably ends in a muted channel.

Consider a workflow checked every 60 seconds with a condition that stays true. That is 60 notifications an hour, 1,440 a day, all carrying the same information as the first one. The rational response is to silence it, and the next real incident arrives in a channel nobody reads.

A better default: fire once, hold, and go quiet while the condition persists. When the metric drops back under the threshold, reset. A second incident is then still audible, because the alert has recovered rather than been suppressed.

Repeating on a fixed interval is worth having for conditions nobody can act on immediately, such as an overnight budget ceiling somebody will address in the morning. It should be a choice you make per alert, not the behavior you get by accident.

Test the threshold before you trust it

Here is the part that is missing from most alerting setups, and it is the one that would prevent the majority of alert fatigue.

You can know whether a threshold is reasonable before you save it. The history is right there. Replay the proposed condition against the last day or week of real runs and count how many times it would have fired.

The answer is immediately actionable:

Zero firings over a week means the threshold is too loose to catch anything, or the metric never moves and you do not need the alert.
Four hundred firings in a day means you have built a noise generator, and you found out in the wizard rather than at 3am.
Two or three firings, on days something actually happened means the number is about right.

Guessing a threshold and waiting a week to learn whether it was sensible is an expensive way to get the same information.

Attach the evidence at firing time

The first question after any alert is why. Answering it should not require a separate investigation.

Store the context at the moment the alert fires, not a link to go and reconstruct it later. For an error alert, that means the failing run ids and the actual error messages. For a cost alert, the per-model breakdown, because "spend doubled" and "spend doubled and it was all one model that got swapped yesterday" are different findings. For an execution count alert, which trigger fired, since that is nearly always the answer.

There is a practical reason to snapshot rather than link. The window has passed. Recomputing it later can genuinely give a different number as data ages out or gets pruned, and an explanation that disagrees with the alert that prompted it is worse than no explanation.

How this works in Heym

The Alerts tab implements the four metric types above, each measured over a window you choose, scoped to one workflow or to everything in your account.

A few specifics worth calling out, because they map onto the arguments above:

The wizard backtests before you save. The review step replays your condition against real history and reports how often it would have fired and the highest value it saw. A threshold that would have fired four hundred times gets corrected in the form.

Cost figures come from the same place as the Traces tab. Token counts and dollar amounts resolve through one pricing table, so a cost alert and the cost page never disagree about the same window. An alert that contradicts your own dashboard is worse than no alert.

Firing once until recovery is the default. Repeating on an interval is available on the same step, as a deliberate choice.

Describe it in plain language and the form fills itself. Writing "warn me if the invoice sync fails more than 5 times in 10 minutes" produces a complete alert definition and jumps to the review step with the guessed fields marked, so you confirm rather than retype. It cannot produce a configuration the system would reject.

A firing can run any workflow. The observed value, threshold, window, and contributing runs arrive as that workflow's input, so Slack, email, and Telegram all work with nodes you already have rather than a separate notification system. One guard applies: an alert cannot notify the workflow it watches, since for an execution count alert each notification would add to the count it measures.

The Chat tab answers questions about them. Asking why a specific alert fired returns the observed value, the threshold, the exact window, and the runs behind it, read from what was recorded at the time.

Where to start

If you are working out how to set LLM cost alerts for the first time, three alerts cover most of the ground:

Execution count, tight window. Catches the runaway trigger, which causes more incidents than it gets credit for.
Error count, medium window. Catches real outages without paging on transient failures.
Spend, daily window. The backstop, as a ceiling rather than a tripwire.

Backtest each one before saving. Add duration alerts once you have a sense of what normal looks like for the workflows you care about, since that is the metric where the baseline varies most between workloads.

The goal is not comprehensive coverage. It is a small number of alerts that are still trusted six months from now.

Working template: Resilient HTTP + Error Handler covers the per-failure half of this, catching a single failing request and notifying on the spot.

Related reading: AI Agent Observability covers the traces and metrics these thresholds are drawn from. AI Agent Cost Optimization covers what to do once a cost alert tells you something is wrong. OpenTelemetry for AI Agents covers exporting the same data to an external observability stack.