eval-runner

Plugin eval-runner in danielrosehill/Claude-Eval-Runner-Plugin. Commands read from the repo's manifest — 1 month ago.

Skills (6)

• create-eval

  Design a custom eval from scratch, or remix an existing benchmark. Use when the user wants to define the eval itself — task framing, dataset composition, scoring rubric, and reporting format — rather than simply wiring up a framework. Produces a fully specified eval definition ready to be run.

• new-workspace

  Provision a new eval-runner workspace on disk. Use when the user wants to start a new evaluation project — scaffolds evals/, datasets/, results/, and docs/ directories, personalises CLAUDE.md, and (by default) creates a GitHub repo.

• publish-dataset

  Publish an eval dataset to Hugging Face Hub (or GitHub as a fallback). Use when the user wants to share the inputs/labels used by an eval — with a dataset card, licensing, splits, and a content hash so downstream runs can verify integrity.

• publish-eval

  Publish an eval (definition + results) so others can reproduce it. Use when the user wants to share an eval publicly — as a GitHub repo, Hugging Face space, or a standalone writeup. Produces a clean, self-contained bundle with README, task spec, rubric, dataset pointer, and a run report.

• run-eval

  Execute an eval defined in the current workspace and capture results with full metadata. Use when the user wants to actually run an eval (one or many SUTs), collect scored outputs under results/, and produce a run manifest so findings are reproducible and comparable over time.

• setup-eval

  Set up an evaluation in the current workspace. Use when the user wants to scaffold a single eval — choosing an existing framework (DeepEval, Inspect AI, OpenAI Evals, lm-evaluation-harness, LightEval, OLMES, Promptfoo, etc.), adapting an existing benchmark, or preparing a clean slate for a custom ev

Slash commands (7)

• /eval-runner:create-eval

  Design a custom eval (or remix an existing benchmark) — task spec, dataset plan, scoring rubric, reporting format.

• /eval-runner:document-eval

  Write up the rationale and findings of an eval into a durable document under docs/.

• /eval-runner:new-workspace

  Provision a new eval-runner workspace (scaffold + optional GitHub repo).

• /eval-runner:publish-dataset

  Publish an eval dataset to Hugging Face Hub (or GitHub), with dataset card, content hash, and license.

• /eval-runner:publish-eval

  Publish an eval (definition + results) to GitHub, Hugging Face Space, or a local bundle.

• /eval-runner:run-eval

  Run an eval from the current workspace, score outputs, and write a run manifest under results/.

• /eval-runner:setup-eval

  Set up an eval in the current workspace — choose a framework, write the brief, and scaffold the eval directory.

Subagents (1)

• eval-engineer

  Autonomous evaluation engineering subagent. Use for multi-step eval work — choosing a framework, designing a custom eval, running it across multiple SUTs, writing findings, and publishing. Coordinates the plugin's skills end-to-end.

Similar plugins