Skip to content
Vernier
Now building: the evaluation layer for frontier AI

Measure intelligence
with precision.

Vernier is the benchmarking and evaluation infrastructure for AI agents and frontier models: on-demand tasks, realistic environments, and reproducible evals at scale.

View platform

EVAL · WEB-AGENTS-V2

LIVE

LEADERBOARD

  • Agent A86.4
  • Model B81.2
  • Agent C74.8
  • Model D69.1

SCORE OVER RUNS

RUN LOG

SAMPLE

  1. 00:12sandbox ready · browser-04
  2. 00:13task web-agents-v2/184 started
  3. 00:14action click · [data-submit]

Built for teams shipping frontier models and agents.

  • Reproducible tasks
  • Sandboxed environments
  • Deterministic scoring
  • Versioned benchmarks
  • Execution traces

Why Vernier

Evaluation is still assembled by hand.

Frontier labs and agent companies need thousands of reproducible tasks. The work is fragmented across spreadsheets, scripts, and one-off sandboxes.

01 / BUILD

Costly to build

A serious eval is thousands of reviewed tasks. Most teams cannot staff that work and still ship the model.

02 / VALIDATE

Hard to validate

Ambiguous items, leaked answers, and brittle graders make a score look exact when it is not. Every task needs a record.

03 / SCALE

Hard to run at scale

Browsers, terminals, and tool sandboxes are slow and expensive. Throughput is an operations problem, not a script.

Platform

Create. Operate. Evaluate.

One system for the tasks, the environments they run in, and the scores you trust.

  1. 01

    Create

    Generate and curate high-quality tasks. Each item is reviewed, versioned, and kept free of leaked answers.

    01 / CREATE

    • web-agents / checkout-flowvalidated
    • code-agents / refactor-142validated
    • tool-use / calendar-conflictin review
    • reasoning / proof-088validated

  2. 02

    Operate

    Run agents in realistic sandboxes: browsers, terminals, apps, and APIs, isolated and reset between trials.

    02 / OPERATE

    sandbox://browser-04
    • navigate https://sandbox.test/checkout
    • click [data-pay]
    • type card •••• 4242
    • observe status 200 · 1.1s

  3. 03

    Evaluate

    Score at scale, compare systems, and track regressions. Every point on the curve has a trace behind it.

    03 / EVALUATE

    • Agent A86.4+1.2
    • Model B81.2−0.4
    • Agent C74.80.0
    • Model D69.1+0.6

Benchmarks

A scale for every kind of work.

Tasks and environments across the categories teams actually measure. Counts below are illustrative.

Coding agents

Edit, test, and repair real repositories. Graded on the diff, the tests, and the path taken.

Tasks: 2,400+

Computer use

Desktop and application control with pixel- and accessibility-level observations.

Environments: 80+

Web browsing

Multi-step browsing in live, resettable sites. Forms, navigation, and extraction.

Environments: 120+

Reasoning and math

Proofs, quantitative problems, and multi-hop questions with checkable answers.

Tasks: 3,100+

Multimodal

Charts, documents, screenshots, and mixed inputs scored against a known result.

Tasks: 900+

Tool use and function calling

Schemas, retries, and side effects. The grader checks the call, not the prose.

Tools: 60+

Long-horizon planning

Tasks that take many steps, with partial credit and a full action trace.

Horizon: 50+ steps

Safety and red-teaming

Policy, misuse, and jailbreak tasks measured with a fixed, reviewable rubric.

Suites: 18

Domain-specific

Finance, legal, and medical workflows, reviewed before they enter a suite.

Domains: 3

Custom enterprise

Your procedures, turned into a private, versioned benchmark your team can rerun.

Private suites

Figures on each tile are illustrative.

Infrastructure

An SDK, not a spreadsheet.

The same run, the same score, the same trace. Built for people who ship agents every week.

Reproducibility

Pinned images, seeds, and tool versions so a run next month matches the run today.

Versioned tasks

Benchmarks move forward without rewriting history. Compare against the revision you shipped.

Parallel execution

Hundreds of isolated trials at once. A full suite finishes in the time one laptop used to take.

Deterministic scoring

Graders are code. The same trace yields the same score, with the rubric attached.

Detailed traces

Every action, observation, and tool call, ready to inspect when a number moves.

evaluate.py
1from vernier import Benchmark2
3bench = Benchmark("web-agents-v2")4run = bench.evaluate(agent=my_agent, parallel=256)5print(run.report())
0+

Tasks in the catalog

target scale

0

Benchmark categories

plus custom suites

0

Parallel evaluations

per run

0+

Environment types

browsers, shells, APIs

Illustrative figures for demo purposes.

Use cases

One platform. Three kinds of teams.

The same infrastructure, pointed at a different question.

Measure every checkpoint against the same scale.

Hold a stable suite across training runs. See where a model advanced, and where it slipped, before the release.

  • Versioned tasks that do not move under you
  • Regression diffs between checkpoints
  • Traces that explain a score

Pricing

Plans, when we open.

No public prices yet. The shape of the product is already clear.

Starter

Coming soon

For a team proving an eval loop.

  • Core benchmark categories
  • Shared sandbox environments
  • Run reports and scores
  • Public task catalog

Scale

Coming soon

For teams running evals on every change.

  • High parallel execution
  • Versioned tasks and regressions
  • Full execution traces
  • Private team workspace

Enterprise

Coming soon

For organizations with their own workflows.

  • Custom benchmark programs
  • Dedicated environments
  • Private task sets
  • Security review and support

Early access

Build the benchmark.
Ship with confidence.

Join the waitlist. We will write when private access opens. Nothing you enter leaves this page.