ai-evals

Plugin ai-evals from marketplace calibre-labs in Calibre-Labs/reforge-ai-evals. Commands read from the repo's manifest — 6 days ago.

Skills (9)

• eval-code

  Write deterministic, code-based evaluator functions for an AI product. Use this skill whenever you need to write evaluators that check structural properties of AI outputs (format, count, presence, schema compliance) without calling another LLM. Also use when auditing existing code-based evaluators f

• eval-guide

  Entry point for eval work. Use when someone asks for help with evals in general, does not know where to start, asks "what should I do next", or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches the request. Load that skill

• eval-llm-judge

  Write LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompt

• eval-rubric

  Write or update the eval rubric for an AI agent or AI feature: the pass/fail criteria that define good behavior, in four groups (outcome, trajectory, governance, experience), plus the release thresholds that belong in the PRD. Use this skill whenever someone asks to "write an eval rubric", "define w

• llm-align

  Analyze alignment between LLM judge scores and human labels in an eval dataset. Use this skill whenever someone wants to evaluate how well an LLM judge agrees with human reviewers, calculate TPR/TNR, investigate disagreements, or improve a scorer prompt. Trigger on phrases like: "calculate TPR/TNR",

• ticket-to-eval

  Strip PII from a customer support ticket or eval trace and convert it into eval dataset rows. Produces two outputs: a regression dataset row (close to the original input, tagged with failure_mode) and a generalized dataset row (abstracted for broader coverage). Works with local CSV files or any eval

• trace-codes

  Turn free-form notes from reading agent traces into trace codes: named, yes/no failure or quality patterns with definitions, examples and counts. Use this skill whenever someone has annotated traces or review notes and asks to "cluster my notes", "create trace codes", "find the failure modes", "do e

• trace-review-app

  Design and build a custom trace review app for one AI product: a single-page tool where the team reads traces, writes notes, labels pass or fail, tags trace codes and exports the results for other skills. Built around that product's own trace shape, rubric and failure modes, and set up so reviewing

• uig

  Build a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone asks to design an eval framework, audit a test set, build a "user input grid", "synthetic query matrix" or "SQM", impr

Similar plugins