Developer ToolsAI Coding

Gauge

The Gauge CLI is free and MIT-licensed, and local runs need no Gauge account; you pay only for the coding agent's own model usage. Hosted Gauge Agents has a free tier with weekly credits and a Pro plan at $499/month for 5,000 credits, according to withgauge.com, checked November 7, 2026. Check official pricing →

Quick Answer

You rewrite the SKILL.md that tells Claude Code how to use your team’s deployment script. It reads better. But did the agent actually get better at deploying, or did you just make the instructions longer? Gauge exists to answer that question. It is an open-source CLI from the team at withgauge.com that runs real coding agents against your Skills, docs, SDK, or CLI, records what they do, judges each run against criteria you write, and compares the results before and after a change.

The CLI is MIT-licensed and runs locally in Docker with no Gauge account. A paid hosted version adds managed compute, scheduled runs, and pull-request checks.

How It Works

A Gauge test has three parts: a task (the prompt the agent receives), inputs (the Skill, docs, package, or files under test, declared in a gauge.json manifest), and criteria (rubrics describing what a good result looks like). Gauge starts a fresh Docker session, gives the agent the task and the configured inputs, and lets it work. Afterward, a coding agent judges the transcript and output against each criterion.

You can store tests in a local SQLite database or as Markdown files in a gauge-cases/ folder, where the frontmatter lists the criteria, which agents to run, and how many samples to take. Those case files are meant to be committed, so a team can review test changes like code.

The useful part comes when you change something. Rerunning the same test snapshots the current inputs as a new batch, and gauge run compare sets a baseline batch against a candidate batch on completion, number of turns, duration, and how each criterion was judged. Gauge also records structured findings, such as where the agent got stuck, which Skill it never used, and what blocked it.

Supported Agents

The CLI supports Claude Code, Codex, Pi, and OpenCode, using your own logins or API keys. Gauge’s website also lists Antigravity for its hosted service. Agent transcripts are normalized across harnesses, so you can run the same case on several agents and see how each handles your instructions. See Claude Code, Codex, and OpenCode.

A Typical Workflow

Here is how a developer might test a change to a release-notes Skill:

  1. Write the Skill or instructions you want the agent to follow.
  2. Define a representative task, such as “write release notes for the changes since the last tag.”
  3. Write acceptance criteria: the notes group changes by type, link each pull request, and leave out internal refactors.
  4. Run the case with several samples per agent (gauge run --local).
  5. Read the judged results and the findings for the runs that failed.
  6. Change the Skill.
  7. Rerun the same case, which records a new batch.
  8. Compare the two batches, then read a few transcripts to check that the change in score reflects a real improvement.

The same loop works for a new SDK quickstart, a rewritten CLAUDE.md, or the error messages in a CLI. Gauge also has preference prompts, which measure which approach an agent chooses when there are several valid options, for example which library it reaches for.

Testing, Watching, and Guarding Against Regressions

Three ideas get blurred in this space, and Gauge covers only some of them:

  • Agent observability records what an agent did during real work, so you can understand and debug it afterward. That is the job of tools like Agent Beacon and of agent runtime observability in general.
  • Agent evaluation measures whether an agent accomplished a defined task, against criteria set in advance. This is Gauge’s core job, a practical form of AI evals.
  • Regression testing checks whether a change broke something that used to work. With fixed cases and batch comparison, Gauge supports prompt regression testing for instructions and Skills.

See Gauge vs Agent Beacon for how evaluation and session history fit together.

Setup

You need Node.js 22.13 or later, Docker running Linux containers, and credentials for the agent you want to test. The README does not list supported operating systems; any machine that meets those requirements should work, which in practice means Docker Desktop on macOS or Windows. Install with npm install -g @withgauge/cli, then gauge local runtime pull to download the runtime image that matches the CLI version. gauge init --local sets up a project, and gauge install-skill teaches your own coding agent how to drive Gauge, so you can ask it to create and run tests for you.

Privacy and Security

Local runs keep results in a .gauge/ folder in your project: a SQLite database, transcripts, and evidence files. Gauge’s docs say to keep that folder private, since transcripts can contain anything the agent saw. The agent’s model calls still go to its provider (Anthropic, OpenAI, or whichever you use), so local mode is not offline.

The CLI sends anonymous telemetry (command name, timestamp, CLI version, and a random installation ID, but not arguments, prompts, or file contents), which you can turn off with gauge telemetry disable or the GAUGE_TELEMETRY_DISABLED=1 environment variable.

Each local session starts fresh in Docker and sees only the task and the inputs you configured. That isolation is the point: an agent under test runs code it wrote, so never point Gauge at a real production environment, a folder full of secrets, or credentials with broad access. Use test API keys and throwaway accounts. Docker containers are a strong boundary for this kind of work but not a perfect one; the AI agent sandboxes guide explains the trade-offs.

Strengths

  • Measures outcomes, not impressions. It answers “did the change help?” with runs and criteria rather than a gut feeling.
  • Uses real agents. Tests run in the same harnesses your developers or users actually use.
  • Repeatable. Fixed cases, multiple samples, and batch comparison make before-and-after testing routine.
  • Free locally. The MIT-licensed CLI needs no account, and test cases can live in your repository.

Limitations

  • Runs cost money. Every sample is a full agent session billed to your subscription or API key. Running enough samples to see a real difference adds up.
  • The judge is an AI. Criteria are scored by a coding agent reading the transcript, so vague rubrics produce unreliable scores. Write specific criteria and spot-check the judgments.
  • Agent output varies. A small difference between two batches may be noise. More samples help; one run proves little.
  • Local mode has fewer inputs. Native binaries, authenticated MCP servers and APIs, scheduled runs, and pull-request checks are listed for the hosted service only.
  • Young project. The repository is small and active, and its release history is short, so expect commands to change.

Pricing

The CLI is free under the MIT license. Local runs use your own agent subscription or API keys, so the main cost is model usage. Hosted Gauge Agents has a free tier that starts with credits and refills to 100 a week, and a Pro plan at $499 a month for 5,000 credits; Gauge says a typical agent-experience run uses about 20 credits. Enterprise pricing is custom. Gauge also sells a separate AI search visibility product, Gauge Chat, which is unrelated to testing coding agents.

For the concepts behind it, see instruction ablation, agent skills, harness engineering, and our guide to CLAUDE.md, SKILL.md, and common pitfalls.

Continue learning

Explore related guides, tools, workflows, and prompts that help you go deeper into this topic.

See how this tool fits into a workflow

Browse step-by-step AI workflows that use ChatGPT, Claude, Gemini, and other tools.

Frequently Asked Questions

What is Gauge?

Gauge is an open-source command-line tool for testing how coding agents use something you built or wrote: an Agent Skill, project instructions, documentation, an SDK, a CLI, or an MCP server. It runs real agents such as Claude Code and Codex on a task in an isolated Docker session, records the transcript, judges the result against criteria you define, and lets you compare runs before and after a change.

Is Gauge free?

The CLI is free and released under the MIT license, and local mode needs no Gauge account. You still pay for the agent runs themselves, through your Claude or Codex subscription or API keys. Gauge also sells a hosted service with managed compute, scheduled runs, and GitHub pull-request checks, billed in credits.

Which coding agents does Gauge support?

The CLI's README lists Claude Code, Codex, Pi, and OpenCode. Gauge's website also lists Antigravity for its hosted service. You use your own logins or API keys for each agent.

Is Gauge an observability tool?

No. Observability tools record what an agent did during real work so you can inspect it later. Gauge runs controlled test tasks and measures whether the agent succeeded against your criteria, which makes it an evaluation and regression-testing tool. It does keep transcripts, but as evidence for the test, not as a log of everyday sessions.

Does Gauge prove my instructions are better?

It gives you evidence, not proof. Agents behave differently from run to run, and an AI judge scores the results against your rubric, so small differences can be noise. Run enough samples per version, keep the task fixed, and read the transcripts before trusting a change in score.

Last updated: