Agent evaluations for Nuxt coding tasks, powered by @vercel/agent-eval.

pnpm install
cp .env.example .env   # requires VERCEL_OIDC_TOKEN and AI_GATEWAY_API_KEY

Runs agent evaluations with memoization. Only runs (model, eval) pairs that haven't been completed yet.

pnpm run eval                          # Run only missing pairs
pnpm run eval -- claude-opus-4.6       # Run a specific experiment
pnpm run eval -- --force               # Re-run everything
pnpm run eval:smoke                    # Run smoke test (1 eval per experiment)
pnpm run eval:dry                      # Preview what would run

Exports clean results to agent-results.json.

pnpm run export-results                          # Export from all experiments
pnpm run export-results -- claude-opus-4.6       # Export specific experiment

Each eval is a self-contained Nuxt project in evals/. Most evals provide broken or suboptimal starter code that the agent must fix — the prompt describes a symptom without revealing the solution.

evals/nuxt-000-fix-data-fetching/
├── PROMPT.md          # task given to the agent
├── EVAL.ts            # vitest assertions (withheld from the agent)
├── package.json       # Nuxt project manifest
├── nuxt.config.ts
├── tsconfig.json
├── eslint.config.mjs
├── server/
│   └── api/
│       └── greeting.ts
└── app/
    ├── app.vue
    └── pages/
        └── index.vue  # broken starter code the agent must fix

1. Create a directory under evals/ (e.g., evals/nuxt-015-my-eval/)
2. Add PROMPT.md with a vague, symptom-based task description (don't reveal the solution)
3. Add broken or suboptimal starter code in app/ for the agent to fix
4. Add EVAL.ts with vitest assertions that check for the correct fix and reject anti-patterns
5. Add package.json with "type": "module" and "build": "nuxt build"
6. Run pnpm run eval — it will automatically run the new eval for all models

1. Create a config in experiments/ (e.g., experiments/my-model.ts)
2. Add the display name to MODEL_NAMES in scripts/export-results.ts
3. Run pnpm run eval — it will automatically run all evals for the new model

After running evals:

1. Export results: pnpm run export-results
2. Copy to the nuxt.com repo:

cp agent-results.json <path-to-nuxt.com>/agent-results.json

3. Commit and deploy the nuxt.com repo

See LICENSE.