Orchestra

Plugin orchestra from marketplace orchestra-skills in understudylabs/orchestra-agent-tools. Commands read from the repo's manifest — 6 days ago.

Skills (43)

• audit-verifier-reliability

  Use when deciding whether a verifier reward is trustworthy for optimization.

• calibrate-difficulty

  Measure per-band model headroom from a local, source-bound run artifact before investing in training.

• capture-evidence

  Use when a developer wants to build an eval from their real LLM app before changing anything — "measure how my app is doing today", "build an eval from my workload", "we have no baseline", "is my current model actually good". Turns the workload into auditable local artifacts (harness, metric, frozen

• check-routing-health

  Use when a developer asks "is Orchestra causing my errors", "which workloads are routed", "is my routing config actually taking effect", "is my provider healthy", "are there 500s on staging", "what's our error rate", "where is my gateway spend going", or wants self-service diagnostics without asking

• compare-model-sweep

  Use when a developer wants to compare candidate models — any mix of local, gateway, or frontier — on the same eval and see quality, latency, cost, and reliability side by side. "Which model should I use", "sweep these models on my benchmark", "compare Gemma vs the frontier on my eval". To stand up a

• compare-trajectories

  Use when you need to know HOW two model runs differ behaviorally on the same tasks, not just THAT one scores higher — per-task trajectory diffing that classifies the gap as persistence/recovery, knowledge, or format/parsing. "why does the bigger model pass these", "is this gap RL-shaped", "diff thes

• curate-trajectories

  Use to turn loose per-task trajectory JSONs (a Lilac export or local rollout corpus) into a queryable, provenance-tracked, contamination-safe dataset that hard-blocks any selection leaking frozen dev/holdout into a train/RL/distill pool. "curate my trajectories", "which rollouts are train-safe", "ex

• design-simulated-environment

  Use to build a simulated, seeded environment (AutomationBench / verifiers style) so any model can run a captured agentic workload end-to-end and be scored on final state — "simulate this workload's tools", "build a validator for these traces", "let a small model attempt the whole task", "score recal

• deslop

  Use when a user asks to "deslop", "de-AI", "make it sound human", remove AI patterns or tropes, clean up AI writing, review prose for authenticity, or draft substantial natural-sounding prose. Applies to papers, articles, memos, reports, newsletters, cover letters, and other edited writing.

• distill-classifier

  Use when a developer wants to replace an expensive frontier model on a classification workload (binary, multi-class, multi-label, or structured extraction) with a fine-tuned open-weight student — "distill this classifier", "can a small model do this tagging job", "the frontier labels these for $X, m

• export-trace

  Use for "get this trace", "show every request in this trace", "export these trace IDs", or "download this workload's captures for a day". Resolves explicit traces or a bounded workload window through customer APIs, exports captures privately, and hands local files to the trace viewer or ingest-trace

• ingest-traces

  Use when a developer already has production LLM traces — a bucket of captures, provider log exports, or gateway capture files — and wants them visualized, turned into local redacted eval sets, or profiled for cost. "Visualize this trace", "ingest my traces", "turn these logs into an eval set", "wher

• inspect-billing-sources

  Use when a developer wants to inspect Anthropic or Claude billing evidence from email, invoices, usage exports, or an authenticated billing website: "scan my Anthropic bills", "check billing emails", "look at the console usage page", "find spend hotspots from receipts". Optional, approval-gated evid

• install-agent-adapter

  Use when a developer wants to install, enable, update, reinstall, remove, or verify Orchestra in a coding agent - "install Orchestra", "add the Orchestra skills", "make Claude/Cursor/Codex/OpenCode/Hermes/Devin see the skills". Chooses the requested agent platform and routes to the local install, re

• install-codex-plugin

  Compatibility shim for Codex installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify the Orchestra Codex plugin - "install Orchestra in Codex", "add the Codex plugin", "make Codex see the skills". Route to install-agent-adapter with platform codex.

• install-cursor-plugin

  Compatibility shim for Cursor installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify the Orchestra Cursor plugin - "install Orchestra in Cursor", "add the Cursor plugin", "make Cursor see the skills". Route to install-agent-adapter with platform cursor.

• install-opencode-plugin

  Compatibility shim for OpenCode installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify Orchestra in OpenCode - "install Orchestra in OpenCode", "add the OpenCode skills", "make OpenCode see the skills". Route to install-agent-adapter with platform opencode.

• install-plugin

  Compatibility shim for Claude Code installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify the Orchestra Claude Code plugin - "install Orchestra in Claude", "add the Orchestra skills", "why can't Claude see the skill". Route to install-agent-adapter with platform

• instrument

  Use when a developer's LLM app is running but has NO traces yet and wants capture flowing in about a minute with no code changes — "instrument my app", "start capturing my LLM calls", "I have no traces, get me some", "turn on tracing for my agent". Detects the provider SDK, redirects it through the

• ladder

  Use to give a developer an immediate local-vs-frontier model comparison before they have their own traces — "what can a local model do", "is a small model good enough", "compare a local model to a frontier model", "the onboarding climb". For comparing many models on a user's own eval, use compare-mo

• latex-paper-polish

  Polish LaTeX papers and PDFs through evidence proofreading, deslop-style prose cleanup, TeX compilation, warning triage, and layout fixes for widows, orphans, overfull boxes, trailing paragraphs, dense tables, captions, URLs, and artifact paths. Use when the user asks to prepare, review, polish, typ

• local-distillation-lab

  Use when a developer wants to fine-tune or distill a local open model on their Mac and see which training method actually moves a captured workload — "fine-tune a local model", "distill the frontier into a small model", "would training close the gap". Compares baseline, rejection-sampled fine-tuning

• lower-anthropic-bill

  Use when a developer wants to cut Claude or Anthropic API spend: "lower my Anthropic bill", "audit my Claude spend", "find prompt cache failures", "why is cache_read zero", "can we move this from Claude to OpenAI or a local model". Audits call sites, tokenizer risk, cache structure, batchability, an

• manage-local-models

  Use to acquire, cache, organize, and explain local open-weight models — "download a model", "what models do I have", "where did the weights go", "free up model disk", "which Gemma/Nemotron should I pull", "how do open models work". Covers where weights come from and live, formats/quantization, gated

• onboard

  Use as the engaging first-run experience right after the Orchestra plugin is installed, or whenever a developer says "get started", "set me up", "I'm new to this", "onboard me", or asks what Orchestra is and where to begin. Confirms the Orchestra sign-in, profiles the machine and goals in one quick

• operate-benchmark-lab

  Use when a coding agent must operate the full benchmark lifecycle over local benchmark dirs — "build a benchmark from my traces and run models on it", "review and calibrate the eval", "queue a prompt experiment", "is an executor running", "read the rigor report". Covers traces → build-benchmark → re

• optimize-agentic-workload

  Use when a developer's agent — a multi-turn tool-calling loop — should get cheaper, faster, or better. "My agent is too slow", "this workflow costs too much", "test a cheaper model in my tool-calling loop", "A/B the policy model". Covers read-only search loops and state-mutating API workflows alike.

• optimize-local-model-compression

  Use when a developer needs to compress a local model for tool-calling workloads — "which quantization should I use", "why does my model emit broken JSON", "how do I get better tool-call fidelity from a 4-bit model", "OptiQ vs QAT vs naive", "the model stopped calling tools after quantization". Cover

• optimize-workload

  Use when a developer has a measured eval and wants the prompt or route improved without retraining — "optimize my prompt against my eval", "close the quality gap with the cheaper model", "squeeze cost without losing quality". Keeps holdout sealed, runs GEPA (automatic prompt evolution) on train/dev

• plan-hosted-run

  Use when a developer wants to run a hosted job — a fine-tune, an RL (reinforcement-learning) run, batch inference, or large-scale trajectory generation — and asks where, how long, and how much. "What would it cost to fine-tune this", "which provider should I use for this RL run", "is this cheaper lo

• prepare-verifier-handoff

  Use when a workload must learn multi-step behavior through hosted reinforcement-learning (RL) training that local optimization cannot deliver, and needs to become partner-ready. "My agent needs RL", "can we train this policy", "package my environment for a training partner", "is this workload ready

• product-knowledge

  Use when a user asks what Orchestra (formerly Understudy) is, how Understudy Desktop works, how local model serving, Fusion sidekick, evals, model candidate results, rollout labs, Product Knowledge, or Orchestra product capabilities should be explained to developers, customers, or agents.

• ramp-and-verify

  Use for "review this rollout", "compare this workload before and after", "make a task-level rollout view", "ramp this route", or "roll this back". Reads captured traffic into a private reproducible comparison, or applies approved staged traffic changes after a route decision. Historical review is re

• recursive-language-model

  Use when a developer wants a small or local model to take over an agentic task a frontier model one-shots — "can a small model do this whole case", "the prompt is too big for the small model", "hill-climb the local model on this task". Decomposes the task into bounded, flat-context steps behind the

• replay-app-harness

  Use when a coding agent has edited a user's LLM app and wants a regression verdict on the frozen benchmark tasks — "did my code change regress the eval", "run my actual app against the benchmark", "author an app-harness.json". Drafts the understudy.app_harness.v1 sidecar from the user's repo, queues

• run-local-model-lab

  Use when a developer wants to stand up and run a local model on Apple Silicon against their real workload — "run this model on my Mac", "is a local model good enough before I pay for hosted". Covers the MLX serving rig, scored real-workload evals, and the route decision. For comparing many candidate

• share-savings

  Use when a developer wants to share how much Orchestra saved them, submit an anonymous "lower my Anthropic bill" result, prepare a leaderboard receipt, or send savings metrics back to Orchestra. Builds a metrics-only payload from value-report or claim artifacts, redacts identity by construction, and

• simulate-before-launch

  Use when a model-level change — a model swap, route change, prompt or playbook edit — is about to ship and the developer wants confidence before prod: "will this work in prod", "run a simulator before I flip traffic", "gate model changes like a pre-commit check". Replays frozen tasks offline, scores

• understand-workload

  Use when a developer wants a captured prompt, trace, or dataset explained before changing anything — "explain this prompt", "what is this trace actually doing", "decompose this workload", "help me understand this dataset before testing models". Builds a shared mental model (purpose, inputs, tools, s

• understudy

  Use when a developer asks a coding agent to improve an LLM app or agent — "make my LLM app cheaper/faster", "raise quality or reliability", "compare models", "pick a model or route". Orchestrates trace → evaluate → optimize (GEPA, automatic prompt evolution) → compare → deploy via worker skills. Not

• use-understudy-gateway

  Use when a developer wants to run inference or evals through the Orchestra gateway — "route my app through Orchestra", "set up my account and keys", "A/B a model on part of my traffic" — or must choose between local provider keys and the gateway route ("should I use my OpenAI key or the gateway"). H

• verify-serving-parity

  Verify identical weights across serving lanes using observed render fingerprints, paired samples, and explicit evidence status.

• watch-logs

  Use when a developer wants an always-on ops watcher over logs, command output, or endpoints — "every five minutes look at my logs and tell me what's wrong", "watch this log file and flag anomalies", "review my events on a schedule without burning tokens". Deterministic hash-gated triggers fire a che

Similar plugins