Evaluation and Release Gating
Test prompts, models, and agent outputs in the Evals tab before changes reach production workflows.
The problem
Prompt and model changes ship on gut feeling. Output quality shifts silently, and regressions surface as user complaints instead of failed checks.
How it runs today
Teams eyeball a few sample outputs in a playground, pick the answer they like, and hope it generalizes to everything else the workflow will see.
With Heym
The Evals tab turns that into a repeatable check: build a suite around a workflow, add test cases, then score several models at once with a judge model you choose.
Inside the Evals tab
Evals is its own page in the Heym dashboard. Open it from the tab bar, go straight to /evals, or jump there from the command palette.
- 1
Build a suite
Name the suite, choose the workflow to evaluate and the credential that runs it, then edit the suite system prompt in place or ask a model to improve it with Optimize Prompt.
- 2
Add test cases
Write input prompts with the output you expect, or use Generate Test Data to create cases from the suite prompt with your selected model.
- 3
Run across models
Select one or more models for a single evaluation, choose Exact Match, Contains, or LLM-as-Judge scoring, and set runs per test, temperature, and reasoning effort. LLM-as-Judge can use a separate credential so the judge is not scoring its own work.
- 4
Read the results
See pass and fail per test case with actual versus expected output, compare model outputs side by side, and reopen any past run with its saved prompt, input, and output snapshots.
A workflow you can start from
Evaluation also works as a workflow when you want it inside a pipeline. The Blind Eval Trio template fans one prompt out to three models, anonymizes the answers, and has a judge score them blind before revealing which model wrote what.
Where control lives
A separate judge credential keeps scoring independent from the model under test, run history preserves prompt, input, and output snapshots for every evaluation, and guardrails plus human review stop failing outputs from reaching users.
Model and prompt decisions backed by recorded, repeatable evaluations instead of instinct.
Built with these Heym capabilities
Common applications
Deployment and integration
- Self-host with Docker or Kubernetes, keeping data on your infrastructure.
- Connect your own model providers: OpenAI, Ollama, vLLM, and more.
- Integrate over HTTP, webhooks, Slack, email, and MCP tools.
- Expose finished workflows as APIs, portals, or MCP servers.
Have a process that fits this pattern?
Show us the workflow, data sources, tools, and human decisions involved. We will help map it to Heym.