zoto-eval-system

Plugin zoto-eval-system from marketplace zoto-agents in zotoio/zoto-agents. Commands read from the repo's manifest — 6 days ago.

Skills (9)

• zoto-advise-evals

  Coverage gap analyser for eval suites. Scans the codebase across five dimensions — trigger-phrase coverage, schema validation, regression baselines, context citation verification, and status checklist completeness — and produces a structured adviser_report with per-target, per-dimension findings and

• zoto-compare-evals

  Cross-run analysis across two or more eval runs driven by **`report.yml`**. Inputs are resolved per run directory under `{evalsDir}/_runs/<ts>/`; per-backend summaries are nested under **`report.static`** and **`report.llm`**. Emits a flat dataset (runs × cases × dimensions) plus the /canvas hand-of

• zoto-configure-evals

  Writes .zoto/eval-system/config.yml from field values pre-collected by /z-eval-configure (command-owned askQuestion). Refuses bundled update.preserveUserAuthoredCases:false or update.writeMetaMarker:false (including nested payload mirrors) before any manifest read or config write. Validates against

• zoto-create-evals

  Scaffolds static pytest and LLM (@cursor/sdk) eval backends into a host repository, generates eval cases with _meta.generated markers for approved skills/commands/agents/hooks, merges eval scripts into package.json, and writes the persistent manifest plus append-only history. Expects the invoking co

• zoto-eval-tooling

  Invocation reference for all eval-system CLI commands. Use this skill when you need to run eval scripts — never reference script paths directly; always invoke via the package.json aliases documented here.

• zoto-execute-evals

  Executes eval runs via the host repo's package.json scripts (pnpm run eval, eval:full, eval:llm, eval:judge). Forwards --model to the LLM runner via both a CLI flag and the ZOTO_EVAL_MODEL environment variable. After each run, calls eval:update --check and appends the drift status to the run report

• zoto-help-evals

  Project-tailored help for the Eval System plugin. Loads plugins/zoto-eval-system/README.md as the source of truth, inspects current project state (.zoto/eval-system/config.yml, manifest.yml, .env*, evals/_runs/, package.json scripts), and composes an answer that is grounded in the chosen README sect

• zoto-judge-evals

  Adversarial judge for eval runs. Identifies weak graders, under-covered assertions, verbosity spikes, and accuracy/confidence regressions. Produces an enriched llm.yml with soft-metric annotations. Does not call askQuestion — handoff to /z-eval-update is expressed as needs_user_input for the command

• zoto-update-evals

  Diff-aware eval updater. Detects drift between covered targets and generated eval cases for skills and central plugin eval files (commands/agents/hooks), classifies changes as critical or non-critical, and preserves user-authored cases verbatim. Apply-mode regeneration re-invokes the LLM analyser pe

Slash commands (13)

• /zoto-eval-system:z-eval-advise

  Pre-hoc eval coverage adviser. Assesses gaps across five dimensions (trigger-phrase, schema, regression, citation, checklist) and recommends actions. Command-owned askQuestion drives multi-turn interaction at two breakpoints (drill-down selection, action recommendations); zoto-eval-adviser subagent

• /zoto-eval-system:z-eval-compare

  Cross-run analysis across two or more runs. When run fragments match multiple `_runs/` folders and the invocation is undifferentiated, the comparer returns `needs_user_input` enumerating candidates — surface that payload verbatim; **do not** run `askQuestion`. Emits flat dataset and /canvas hand-off

• /zoto-eval-system:z-eval-configure

  Writes .zoto/eval-system/config.yml using command-owned askQuestion for every field, then spawns zoto-eval-configurer with pre-collected answers. Diffs the chosen config against the manifest snapshot (discovery_config.static), renders the resulting cleanup_plan via askQuestion, and — only on explici

• /zoto-eval-system:z-eval-create

  Scaffold static (pytest) and LLM (@cursor/sdk) eval backends, generate eval cases with _meta.generated markers for approved skills/commands/agents/hooks, and write the persistent manifest. Command pre-collects target checklists via askQuestion before spawning zoto-eval-generator.

• /zoto-eval-system:z-eval-execute

  Run the eval suite via the host repo's package.json scripts. Command-owned askQuestion when --full lacks CURSOR_API_KEY; zoto-eval-executor returns needs_user_input if unresolved. --model forwards to LLM runner.

• /zoto-eval-system:z-eval-help

  Section-routed help for the Eval System plugin. Command-owned askQuestion for section selection and follow-ups; zoto-help-evals composes a project-tailored answer anchored on the chosen README section, with mandatory `start:end:path` citations back to the README and any project files it inspected. T

• /zoto-eval-system:z-eval-init

  Initialise the Eval System config at .zoto/eval-system/config.yml with a commented template of all defaults.

• /zoto-eval-system:z-eval-judge

  Adversarial post-hoc analysis of the most recent eval run. Command-owned askQuestion for handoff to /z-eval-update; judge subagent surfaces needs_user_input. Enriches llm.yml with a judge block.

• /zoto-eval-system:z-eval-jump

  Jump operators into the evaluator workflow — verifies `.zoto/eval-system/config.yml`, loads commands/z-eval-workflow.md, then applies its Probe, Lifecycle router, and Resolution sections verbatim (read-only; no subagents).

• /zoto-eval-system:z-eval-operator

  Operator jump into the evaluator workflow — verifies `.zoto/eval-system/config.yml`, loads commands/z-eval-workflow.md, then applies its Probe, Lifecycle router, and Resolution sections verbatim (read-only; no subagents).

• /zoto-eval-system:z-eval-start

  Jump operators into the evaluator workflow — verifies `.zoto/eval-system/config.yml`, loads commands/z-eval-workflow.md, then applies its Probe, Lifecycle router, and Resolution sections verbatim (read-only; no subagents).

• /zoto-eval-system:z-eval-update

  Diff-aware eval updater. Re-invokes the LLM analyser per drifted primitive, dispatches per-framework / per-strategy regeneration through hard-coded user-case preservation gates, and appends a manifest snapshot. Command-owned askQuestion per change in apply modes; subagent uses resume. The `_meta.gen

• /zoto-eval-system:z-eval-workflow

  Interactive evaluator entry point — command-owned askQuestion maps the operator’s current lifecycle stage to the correct Eval System slash command (configure, scaffold, drift, execute, judge, compare, advise, help) without spawning subagents.

Subagents (8)

• zoto-eval-adviser

  Assesses eval suite coverage gaps across five dimensions (trigger-phrase coverage, schema validation, regression baselines, citation verification, checklist completeness). Reads skill/command/agent sources and their evals.json files — never reads run artefacts or modifies eval files. Does not call a

• zoto-eval-analyser-subagent

  System prompt for the LLM-driven primitive analyser invoked by `pnpm run eval:analyse`. Not surfaced to humans — driven by the eval-system stamp/update flows via @cursor/sdk. Produces a strict JSON AnalyserPayload (templates/schema/analyser-payload.schema.json) per primitive (skill / command / agent

• zoto-eval-comparer

  Cross-run analysis that emits a flat dataset and a /canvas hand-off instruction. The host agent is expected to invoke Cursor's built-in /canvas tool to render the comparison. Skill never renders charts itself. Drilldown opens per-case log files. Does not call askQuestion — when fragments (e.g. `2026

• zoto-eval-configurer

  Writes .zoto/eval-system/config.yml using the zoto-configure-evals skill. Presents enum-backed fields only through answers pre-collected by the invoking command; never calls askQuestion. Reads the manifest snapshot via plugins/zoto-eval-system/engine/manifest-snapshot.ts, emits a cleanup_plan valida

• zoto-eval-executor

  Executes eval runs via the host repo's package.json scripts. Forwards --model to the LLM runner via both a CLI flag and the ZOTO_EVAL_MODEL environment variable. Gates the LLM backend on CURSOR_API_KEY and --full. After every run, calls eval:update --check and appends the drift status to llm.yml as

• zoto-eval-generator

  Plans and scaffolds the eval suite for a repository. Uses zoto-create-evals and zoto-configure-evals skills to lay down both pytest and LLM (@cursor/sdk) backends, generate per-target eval cases with _meta.generated markers, and write the persistent manifest plus append-only history. Does not call a

• zoto-eval-judge

  Adversarial eval judge. Reads the most recent llm.yml and per-case logs, flags weak graders, under-covered assertions, verbosity spikes, and low-confidence cases. Writes a judge block to llm.yml. Does not call askQuestion — returns needs_user_input when handoff to /z-eval-update requires user approv

• zoto-eval-updater

  Diff-aware eval updater. Uses zoto-update-evals to detect drift between covered targets and generated eval cases, re-invokes the LLM analyser per drifted primitive, and dispatches per-framework static regeneration plus the single unified LLM eval regeneration (`regenerateLlm`) through hard-coded use

Similar plugins