Test your AI agent before it breaks a real workflow.
Record every run, catch failures with reasons, compare prompts and models — and know your success rate before you ship.
The diagnosis your team can't produce by watching runs by hand.
One context manager. Every run recorded.
Wrap any Python agent — LangChain, browser-use, or plain code — and AgentQA captures the trace: steps, tool calls, screenshots, cost, latency. Define test cases with expected behavior and forbidden actions, then run the suite on every change.
- 02Deterministic rules score what rules can score — forbidden tools, budgets, formats
- 03An LLM judge handles policy adherence and unsupported claims
- 04Uncertain cases go to human review — never guessed
import agentqa
with agentqa.track_run(
agent_name="sales_research_agent",
task_input=task,
) as run:
result = my_agent.run(task)
run.set_output(result)
# every step, tool call, screenshot,
# cost and verdict — recordedKnow your success rate, not your vibe
Run your agent against a real test suite and get pass/fail per case — with the failure reason attached, from hallucinated facts to wrong tool calls.
Catch regressions before customers do
Every prompt tweak and model update runs against the same suite. See exactly what got better, what broke, and what it costs per successful task.
Replay every failure
Full traces — steps, tool calls, screenshots — so “what happened?” takes one click, not an afternoon of archaeology.
Verdicts with reasons, not just logs
Observability tools show you the trace. AgentQA rules on whether the job was done correctly — and classifies every failure against a library of real agent failure patterns.
Simple pricing
Start free, upgrade when your test suites do. Payments are handled by Polar — taxes included, cancel anytime.
Free
$0
Kick the tires on one agent.
- 1 project
- 50 judge evaluations / day
- Full trace capture
- Shareable reports
Starter
Recommended$99/mo
For a team shipping its first agents.
- 3 projects
- 500 judge evaluations / day
- Everything in Free
- Email support
Team
$499/mo
For agencies running many workflows.
- 10 projects
- 2,000 judge evaluations / day
- Everything in Starter
- Priority support
Agent reliability audit
Shipping agents for clients? We test one agent workflow end to end — 20+ cases, failure classification, and a shareable report with hard before/after numbers. $1,000–$5,000, one to two weeks.
Book an audit