ai-eval

Plugin ai-eval in danielrosehill/AI-Eval. Commands read from the repo's manifest — 8 days ago.

Skills (34)

• add-prompts

  Add test prompts to an existing eval's prompt set by delegating the writing to a Haiku subagent at once, then validating, merging and republishing in the parent. Use when the user says "add some prompts", "write more prompts for this", "extend the prompt set", "add a series on X", or "we need more c

• agent-eval

  Evaluate an AI agent or tool-using model - tasks with checkable end states, sandboxing, and repeat-attempt reliability. Use when the user says "evaluate my agent", "test tool calling", "how reliable is this agent", "which model is best for agents", or "does it complete the task".

• analyse-findings

  Write the analytical findings memo for an eval run. States what the numbers show, with evidence tiers, tests the author's hypotheses (including those that didn't hold), and covers limitations. An adversarial check against the raw data runs before anything is kept. Writes findings/<run_id>/findings.m

• create-eval

  Design a custom eval from scratch, or remix an existing benchmark. Use when the user wants to define the eval itself — task framing, dataset composition, scoring rubric, and reporting format — rather than simply wiring up a framework. Produces a fully specified eval definition ready to be run.

• define-objective

  Pin down what an eval is for before any model is chosen - the decision it informs, what a win looks like, who judges, and what's out of scope. First step of model selection. Use when the user says "I want to find the best model for X", "I'm running an eval for X", "help me work out what I'm testing"

• define-workload-requirements

  Turn a specific workload into concrete model requirements - real context load, output length, modalities, capabilities, cost per output - with each requirement traced to a feature of the work. The foundation for intelligent model selection. Use when the user says "what does this workload need", "wha

• deploy-arena

  Add human A/B rating to an eval: a blinded pairwise arena (local, or hosted on a private Hugging Face Space) whose ratings are kept in a private dataset and folded into the report alongside the LLM judge. Use when the user says \"I want to rate these myself\", \"set up the arena\", \"human eval\", \

• draft-article

  Draft a blog post, journalistic article or press-style summary from an eval's findings memo, for a general or specialist audience. Writes findings/<run_id>/article-<audience>.md and hands publishing to the blog skills. Use when the user says "write a blog post about the eval", "turn the findings int

• draft-test-prompts

  Write an eval's test prompts as data/prompts.jsonl with a dataset card, publish them as the eval's Hugging Face dataset, and link it from the README. Use after setup-eval, or when the user says "write the prompts", "publish the prompt set", "push the test set to HF", or "add prompts to the eval".

• full-run

  Full run of an authored eval — every prompt against a model panel, with an LLM judge scoring every answer against the rubric. Adds the harness to the eval repo if there isn't one (generate → judge → report, resumable, via OpenRouter) and writes runs/<run_id>/. Use only when the user explicitly asks

• go-public

  Pre-publication review for an eval. Use it to take an eval that was explicitly created private public (in whole or in part), or to review a public eval before its results are promoted or cited. Checks method, model selection, git history for credentials, dead links and logged-out access, flips any p

• image-gen-eval

  Evaluate generative image models for a real job - model sourcing, per-image pricing, prompt set design, blind human grading, and where automated checks help or mislead. Use when the user says "evaluate image models", "which image model should I use", "compare image generators", "image eval", or "tex

• informal-image-eval

  Informal micro-round of an image-generation eval on fal - refresh the list of current fal image models and prices, pick 3-5 cost-effective candidates, run one or two text-to-image prompts with two tries each (about 20 images, well under $1), and put the images in the eval's annotation Space for huma

• install-eval-cli

  Install an evaluation framework CLI on Linux (Inspect AI, DeepEval, promptfoo, lm-evaluation-harness, LightEval, EvalScope) and confirm it runs. Use when the user says "install promptfoo", "set up Inspect", "I want to use a framework", "which eval CLI should I use", or before setup-framework-eval.

• new-workspace

  Provision a new ai-eval workspace on disk. Use when the user wants to start a new evaluation project — scaffolds evals/, datasets/, results/, and docs/ directories, personalises CLAUDE.md, and (by default) creates a GitHub repo.

• package-results

  Turn an eval run into shareable outputs beyond the raw dataset. Produces tidy CSV tables with n and confidence intervals, charts, an excerpt sheet of verbatim answers, and an optional results brief PDF, all in exports/<run_id>/. Use when the user says "package the results", "export the results", "ma

• probe-openrouter

  Probe OpenRouter for the current state of its catalogue - snapshot every model with context, output limits, prices, modalities and parameters, list what's new, check the API key, inspect who serves a model, and confirm models actually answer. Use when the user says "what's on OpenRouter", "what's ne

• publish-dataset

  Publish an eval dataset to Hugging Face Hub (or GitHub as a fallback). Use when the user wants to share the inputs/labels used by an eval — with a dataset card, licensing, splits, and a content hash so downstream runs can verify integrity.

• publish-eval

  Publish an eval (definition + results) so others can reproduce it. Use when the user wants to share an eval publicly — as a GitHub repo, Hugging Face space, or a standalone writeup. Produces a clean, self-contained bundle with README, task spec, rubric, dataset pointer, and a run report.

• publish-run

  Push an eval run's raw data (responses, judge scores, human ratings) to the eval's <Name>-Eval-Runs Hugging Face dataset (public by default), one config per run, and add the results to the README. Use after run-eval, or when the user says "publish the run", "push the results to HF", or "make a datas

• rag-eval

  Evaluate a retrieval or RAG pipeline - separating retrieval from generation, building a question set from real documents, and comparing generators with retrieval held fixed. Use when the user says "evaluate my RAG", "test retrieval", "is the chatbot finding the right documents", "which embedding mod

• register-eval

  Keep the user's eval indexes in step with a public eval: add it or update its stage in whichever index, registry or scorecard the user keeps, and retire the plan file it graduated from. Called by setup-eval, draft-test-prompts, publish-run and go-public on stage changes; also when the user says \"in

• regression-eval

  Guard a working prompt, model or pipeline against regressions - a fixed test set that runs on every change and fails when quality drops. Use when the user says "make sure this doesn't break", "test prompt changes", "evals in CI", "can I safely swap the model", or "did my change make it worse".

• run-eval

  Run an eval the wide-and-shallow way - many candidate models, about two outputs each - under a hard spend cap, and report which models are worth a closer look. The default way to run. Use when the user says "run the eval", "run it", "test these models", "do a test run", "try it on a bunch of models"

• run-framework-eval

  Execute an eval defined in the current workspace and capture results with full metadata. Use when the user wants to actually run an eval (one or many SUTs), collect scored outputs under results/, and produce a run manifest so findings are reproducible and comparable over time.

• setup-agent-eval-space

  Set up an eval that needs no framework - the AI agent is the harness, following a written procedure to call the models, read the outputs and judge them. Use when the user says "just do it with Claude", "no framework", "keep it simple", "have the agent run it", or for a small eval where a framework w

• setup-annotation-space

  Set up a private Hugging Face Space for grading an eval's outputs by hand - blind to the model, with a rubric from the eval, saving to a private dataset and flowing back into the report. Works for text, image and audio. Use when the user says "set up an annotation space", "I want to grade these myse

• setup-eval

  Start a new eval of your own, one repo per eval. Checks whether one already exists, then creates a public GitHub repo with the design and rubric, reserves a public Hugging Face dataset for the prompts, and registers it in the public indexes. Private only when the user asks for it. Use when the user

• setup-framework-eval

  Set up an evaluation in the current workspace. Use when the user wants to scaffold a single eval — choosing an existing framework (DeepEval, Inspect AI, OpenAI Evals, lm-evaluation-harness, LightEval, OLMES, Promptfoo, etc.), adapting an existing benchmark, or preparing a clean slate for a custom ev

• setup-human-feedback-space

  Set up a private Hugging Face Space that collects human preference feedback as training data - chosen and rejected pairs for RLHF, DPO or reward models - rather than for scoring an eval. Use when the user says "human feedback", "RLHF", "collect preferences", "preference data", "DPO dataset", "train

• setup-repo-eval-space

  Add an eval space inside an existing project repo, so evals live next to the code and prompts they test. Use when the user says "put the evals in this repo", "add an eval folder here", "set up evals for this project", or when start-eval routes here. For a standalone citable eval use setup-eval; for

• shortlist-candidates

  Draw up the list of candidate models for an eval from the OpenRouter catalogue - filter on the workload's hard requirements, then pick a wide, varied field with a reason for each. Use when the user says "which models should I test", "draw up candidates", "build the model list", or after define-workl

• speech-eval

  Evaluate speech models - speech-to-text with word error rate on the user's real audio, and text-to-speech with blind human listening. Use when the user says "evaluate ASR", "which transcription model", "WER", "test speech recognition", "compare TTS voices", or "which text-to-speech is best".

• start-eval

  The front door for evaluating anything - takes the user through an eval on their project, working out what already exists and routing to the right skills. Use when the user says "I want to do an eval on this project", "help me evaluate this", "set up evals", "how should I test this", "which model sh

Slash commands (7)

• /ai-eval:create-eval

  Design a custom eval (or remix an existing benchmark) — task spec, dataset plan, scoring rubric, reporting format.

• /ai-eval:document-eval

  Write up the rationale and findings of an eval into a durable document under docs/.

• /ai-eval:new-workspace

  Provision a new ai-eval workspace (scaffold + optional GitHub repo).

• /ai-eval:publish-dataset

  Publish an eval dataset to Hugging Face Hub (or GitHub), with dataset card, content hash, and license.

• /ai-eval:publish-eval

  Publish an eval (definition + results) to GitHub, Hugging Face Space, or a local bundle.

• /ai-eval:run-framework-eval

  Run an eval from the current workspace, score outputs, and write a run manifest under results/.

• /ai-eval:setup-framework-eval

  Set up an eval in the current workspace — choose a framework, write the brief, and scaffold the eval directory.

Subagents (1)

• eval-engineer

  Autonomous evaluation engineering subagent. Use for multi-step eval work — choosing a framework, designing a custom eval, running it across multiple SUTs, writing findings, and publishing. Coordinates the plugin's skills end-to-end.

Similar plugins