Evaluation and Release Gating

Test prompts across models and check agent tool calls, latency, and cost in the Evals tab before changes reach production.

Discuss your use case View the template

The problem

Prompt and model changes ship on gut feeling. Output quality shifts silently, and regressions surface as user complaints instead of failed checks.

How it runs today

Teams eyeball a few sample outputs in a playground, pick the answer they like, and hope it generalizes to everything else the workflow will see.

With Heym

The Evals tab turns that into a repeatable check: prompt suites score several models on your test cases, and agent suites check tool calls, latency, and cost.

Inside the Evals tab

Evals is its own page in the Heym dashboard. Open it from the tab bar, go straight to /evals, or jump there from the command palette. It holds two kinds of suite: prompt suites, which test a system prompt across models, and agent suites, which check how an agent behaves on its real runs.

1
Build a prompt suite
Choose Prompt suite, pick the workflow and the credential that runs it, and edit the system prompt in place or improve it with Optimize Prompt. Write test cases with the output you expect, or create them with Generate Test Data.
2
Run across models
Select one or more models for a single evaluation, choose Exact Match, Contains, or LLM-as-Judge scoring, and set runs per test, temperature, and reasoning effort. LLM-as-Judge can use a separate credential so the judge is not scoring its own work.
3
Read the results
See pass and fail per test case with actual versus expected output, compare model outputs side by side, and reopen any past run with its saved prompt, input, and output snapshots.
4
Check an agent
Choose Agent suite, pick a workflow and its Agent node, and read its baseline from the last 10 runs. Mark each tool Must call, Must not call, or Either, including MCP tools, sub-agents, sub-workflows, and skills, and set upper bounds for latency, cost, tokens, and tool calls.
5
Run the agent and compare
Run the workflow up to 20 times with inputs you edit or have a model generate, or score the agent’s latest runs. Results show the pass rate, how often each check passed, the numbers next to the baseline, and each failed run with its reason and a link to its trace.
6
Ask from Chat or MCP
Ask the Chat tab, or call heym_chat from an MCP client, to create, change, and run prompt and agent suites. It writes the test cases and run inputs, sets the baseline you ask for, and replies with a link to each suite in Evals.

See it in Heym

A short walkthrough recorded in the product.

Heym Evals: Test Every Prompt Against Every Model — Building a suite, generating test cases, running several models side by side, and reading judge scores, accuracy history, and exports in the Evals tab.

A workflow you can start from

Evaluation also works inside a pipeline, with a Decision Model as the judge. The Decision Model RAG Reranker template retrieves up to eight chunks, has a RelevanceJudge node score each one against the question on a 0 to 3 rubric, and drops anything below 2. The answer model runs only when evidence passes that gate, so a weak retrieval ends in a request to refine the question instead of a guessed answer.

Prompt or agent suiteRuns and checksScored resultsApproved configuration ships

Decision Model RAG Reranker

Sticky Note

Before you run

Select an existing vector store on RetrieveCandidates.
Match dbType to Qdrant or pgvector.
Add a Decision Model credential to RelevanceJudge.
Add an LLM credential to GroundedAnswer.
Ingest your own documents before searching.

Try a direct answer, a tangential chunk, and an unrelated chunk.

Sticky Note

Relevance rubric

0: Unrelated.
1: Same topic, no useful answer.
2: Useful partial evidence.
3: Direct evidence for the answer.
Keep scores >= 2, sort descending, take 3.
No qualifying evidence means no LLM call.
One decision request per candidate chunk.

Question

RetrieveCandidates

InitEvidence

CandidateLoop

loop

done

RelevanceJudge

ScoredChunk

CollectEvidence

SelectEvidence

HasEvidence

true

false

GroundedAnswer

gpt-4.1-mini

Answer

NoEvidence

Sticky Note

Live run checklist

Test both exits around HasEvidence before going live.
Tune the schedule, wait, or batch size first so repeated runs stay predictable.
Keep node labels stable if you update expressions that reference earlier outputs.

Open this template, inspect every node, and import it

Where control lives

A separate judge credential keeps scoring independent from the model under test, agent suites compare every run with a saved baseline and link each failure to its trace, and guardrails plus human review stop failing outputs from reaching users.

Model, prompt, and agent decisions backed by recorded, repeatable evaluations instead of instinct.

Built with these Heym capabilities

Eval suites per workflow

Agent suites with tool checks

Latency, cost, and token limits

Generated test data and run inputs

Multi-model evaluation runs

Exact match, contains, and LLM judge scoring

Separate judge credential

Suites from Chat and MCP

Guardrails

Human review

Common applications

Model selection across candidatesPrompt regression testingAgent tool-call regression checksLatency and cost budgets per agentHuman-in-the-loop scenario coverageJudge-scored answer qualityBlind comparison of competing models

Deployment and integration

Self-host with Docker or Kubernetes, keeping data on your infrastructure.
Connect your own model providers: OpenAI, Ollama, vLLM, and more.
Integrate over HTTP, webhooks, Slack, email, and MCP tools.
Expose finished workflows as APIs, portals, or MCP servers.

Have a process that fits this pattern?

Show us the workflow, data sources, tools, and human decisions involved. We will help map it to Heym.

Discuss your use case Explore workflow templates