orq-mcp

Plugin orq-mcp from marketplace orq-claude-plugin in orq-ai/assistant-plugins. Commands read from the repo's manifest — 2 days ago.

Skills (21)

• skill-tests

  Test orq skills in this checkout, or set up tests for a skill that has none. Runs factual drift checks for tools, CLI commands, SDK calls and URLs named in SKILL.md, plus invocation and behavioural evals in real Claude Code and OpenCode sessions through `orq launch`. Reports whether the right skill

• evaluatorq

  Write and run evaluatorq evaluation scripts (Python or TypeScript) for a single agent or deployment — custom scorers, built-in evaluators, LLM judges and juries, pairwise preference judging, and evaluation driven by datasets, past experiments or production traces. For CLI workflows, use the companio

• orq-analyze-traces

  Analyze a live agent, deployment, or local agent from its production traces — relay its configuration and terminal states, then build a failure taxonomy by open coding and axial coding, and write it to an error-analysis file other skills read. Use when debugging agent or pipeline quality, when you h

• orq-build-agent

  Design, create, and configure orq.ai Agents with tools, instructions, knowledge bases, and memory stores. Use when building new agents, attaching KBs or memory, writing system instructions, selecting models, or setting up RAG pipelines. Do NOT use for debugging existing agents (use orq-analyze-trace

• orq-build-evaluator

  Create validated LLM-as-a-Judge evaluators following best practices — binary Pass/Fail judges by default, plus numeric and categorical, all validated against human labels for measuring specific failure modes. Use when you need to automate quality checks, build guardrails, or measure a specific failu

• orq-cli

  Drive the `orq` command-line interface — check the install, authenticate, select a workspace, and run read and write commands against any orq.ai resource (traces, agents, deployments, evals, prompts, datasets, projects, skills). Use when a task needs shell access to orq.ai, when a script or CI job m

• orq-compare-agents

  Run cross-framework agent comparisons using evaluatorq from orqkit — compares any combination of agents (orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK) head-to-head on the same dataset with LLM-as-a-judge scoring. Use when comparing agents, benchmarking, or wanting side-by-side evaluat

• orq-create-skill

  Build or update an agent skill from an API, CLI, or MCP surface. Use when the user wants to document a new capability surface as a skill, or when an existing skill's contract is stale and needs re-verification. Do NOT use when editing an existing skill without re-probing the surface, or when documen

• orq-evaluator-alignment

  Align, calibrate, or improve an existing LLM-as-a-judge (orq evaluator) so its verdicts match human judgment — boolean, categorical, or numeric judges. Use when the user wants to "align my evaluator", "improve my eval", "my judge keeps changing its mind", "find ambiguous cases", or "annotate an eval

• orq-generate-synthetic-dataset

  Generate and curate evaluation datasets — structured generation via dimensions-tuples-NL, quick from description, expansion from existing data, plus dataset maintenance through deduplication, rebalancing, and gap-filling. Use when creating eval data, expanding test coverage, or cleaning datasets. Do

• orq-improve-agent

  Improve an underperforming orq agent, deployment, or local agent — rewrite its instructions against a structured prompting framework, or move a configuration knob, grounded in the error-analysis file orq-analyze-traces writes. Use when a prompt needs improvement, when a config knob is wrong (truncat

• orq-invoke-deployment

  Invoke orq.ai deployments, agents, and models via the Python SDK or HTTP API. Use when a user wants to call a deployment with prompt variables, invoke an agent in a conversation, or call a model directly through the AI Router. Do NOT use for creating or editing deployments/agents (use orq-improve-ag

• orq-manage-skills

  Manage orq.ai Skills (the platform entity, formerly called Snippets) end-to-end — list, get, create, update, and delete Skills, plus authoring guidance (display name, description, tags, project scoping, path placement), and how Skills get consumed (the `{{skill.<display_name>}}` template placeholder

• orq-quickstart

  Onboarding for orq.ai in any coding agent — get an API key, register the orq.ai MCP server for the harness you are running in, prove the connection with a real workspace read, show a workspace snapshot, list the installed orq skills, and route the user to a first task. Use when the user is new to or

• orq-recommend-evaluators

  Use when an orq agent or deployment has no or few evaluators and someone asks which ones it should have ("what should I evaluate", "which evals does this agent need", "suggest evaluators"), including a new agent with no production traffic yet. Do NOT use to build and validate one specific judge in d

• orq-red-team

  Invoke the evaluatorq red teaming CLI to run adversarial attacks against deployed agents or static datasets. Use when asked to "run a red team", "red team this deployment", "test my agent for vulnerabilities", "OWASP red team", or "check how my agent handles adversarial inputs". Do NOT use when you

• orq-review-agent

  Health-check an orq agent's setup without changing it, with or without production traffic: instructions, tools, skill references, evaluators, model choice and run health, checked against the suite's best practices, each finding with a suggestion, in one HTML report. Use when someone asks to "review

• orq-run-experiment

  Create and run orq.ai experiments — compare configurations against datasets using evaluators, analyze results, and generate prioritized action plans. Use when evaluating LLM agents, deployments, conversations, or RAG pipelines end-to-end. Do NOT use without a dataset and evaluators. Do NOT use for c

• orq-setup-observability

  Set up orq.ai observability for LLM applications. Use when setting up tracing, adding the AI Router proxy, integrating OpenTelemetry, auditing existing instrumentation, or enriching traces with metadata.

• orq-shared

  Reference bundle shared by the other orq skills — the verified `orq` CLI trace query contract, the run-key preflight that resolves an agent or deployment key before any call, and the rubrics an agent is built, reviewed and improved against. Read the file another skill points you at; do not invoke th

• orq-simulate-agent

  Run multi-turn agent simulations using evaluatorq's first-class simulation primitives (`simulate()`, `generate_and_simulate()`, `wrap_simulation_agent()`). Drive an agent under test with a simulated user LLM, scored by a built-in JudgeAgent that decides per turn whether the goal was achieved or rule

Slash commands (5)

• /orq-mcp:analytics

  Show workspace analytics — requests, cost, tokens, errors, top models, and drill-down trends

• /orq-mcp:models

  List available AI models and their capabilities

• /orq-mcp:orq-manage-skills

  Manage orq.ai Skills — list, get, create, update, retire (tag as retired), or delete Skills (the platform entity, formerly Snippets) and find the prompts/agents that reference them

• /orq-mcp:traces

  Query and summarize traces with filters — debugging entry point before orq-analyze-traces

• /orq-mcp:workspace

  Show a workspace overview — agents, deployments, prompts, datasets, experiments, projects, and knowledge bases

Subagents (1)

• orq-assistant

  orq.ai workspace assistant — routes to skills and commands for building, evaluating, and monitoring LLM pipelines

MCP servers (1)

• orq-workspace

  connects to: https://my.orq.ai/v2/mcp

Similar plugins