claudecodex

production-evals-monitoring

Watch an AI feature's answer quality in production (samples, judges, alerts, drift). Use before go-live, for a post-market monitoring plan, or when a pass rate drops and someone wants to roll back.

1 on GitHubv1.1.0updated 2 days ago
/

Plugin production-evals-monitoring from marketplace ai-pm-skills in alexandre-darmon/ai-pm-skills. Commands read from the repo's manifest — 2 days ago.

What it installs

Skills (29)

  • agent-design-and-tooling

    Design an agent''s loop, tools, permissions, stop conditions and traces from its autonomy levels, before any framework. Use for "make it an agent", a new tool or permission, or a framework debate.

  • agent-evals

    Test an agent that calls tools or acts on systems and gate its autonomy on measured reliability. Use for "how do we test our agent?", a level change, a new tool, or "it says done, nothing changed".

  • ai-act-classification

    Classify an AI use case under the EU AI Act (role, risk category, obligations, questions for counsel). Use for a new AI use case, an autonomy level change, or a vendor's AI Act claim.

  • ai-adoption-plan

    Plan and prove adoption of an AI tool on outcomes: correct results per user, overrides, over-trust test, training by role and level, art. 4 evidence. Use when go-live nears or use stalls.

  • ai-data-requirements

    Specify what an AI feature reads and writes. Use on "what data do we need", a DPO asking what is processed, or to set up or review a RAG store: stale answers, mixed sensitivity, unowned docs.

  • ai-feasibility-spike

    Use when someone wants to prototype or spike an AI feature or agent on the real model before a PRD, or has spike results to read, write up as a go note or offer as validation evidence.

  • ai-incident-playbook

    Prepare or run the response to an AI incident: severity, containment, rollback owner, reporting clocks, the regression case it adds. Use at L3+ go-live, after a quality alert or in an incident.

  • ai-interaction-patterns

    Decide what users see of an AI answer: sources, a confidence label or score, correction, refusal, hand-off per autonomy level. Use for assistant UX, a "% sure" badge, or reviewers who accept all.

  • ai-opportunity-assessment

    Decide in one page if an AI idea deserves a PRD, in which form, and what to ask the sponsor: a dated pilot, stop conditions, an evidence plan. Use for "should we do this with AI?" or an AI pitch.

  • ai-roi-quantification

    Compute a deployed AI product''s ROI with the five-line model, a baseline and sourced unit values. Use for a budget renewal, a CFO or comex review, "what did it return?" or a scaling decision.

  • ai-vendor-evaluation

    Choose or renew a bought AI product or vendor (copilot, chatbot, document AI) on your own bake-off, cost per correct outcome and contract terms. Use for an RFP, a vendor demo or claim, a renewal.

  • assistant-kpis-cost-per-outcome

    Define an AI assistant's or agent's KPIs (quality by domain, adoption, cost per correct outcome). Use when asked for its dashboard, or when usage grows and nobody knows if it helps.

  • build-evals

    Build or review AI eval sets, graders, thresholds. For an eval stub, disputed quality, which pipeline step fails, a costly class at N of M just over its signed bar. Not to pick between scored models.

  • conversation-analysis

    Classify an AI assistant's conversations by use case, resolution and sentiment, PII masked. Use to build or check the taxonomy, pick the classifier, see what people use it for, or act on the results.

  • error-analysis

    Turn an AI feature''s traces or reviewers'' notes into counted, priced failure modes before any metric or judge. Use for unexplained bad answers, "what should we measure?", or notes to group.

  • llm-feature-prd

    Write the PRD for an LLM feature with autonomy level per step, failure paths, eval plan and guardrails. Use when asked for the PRD of anything that generates, extracts, classifies or acts.

  • llm-judge-calibration

    Check if an LLM judge can be trusted on a failure mode and correct the pass rate it reports. Use for "can we use LLM-as-judge?", a judge's agreement with experts, a judge change or drift.

  • model-card-ai-registry

    Write an AI system''s model card and registry entry as its eval record: results per class with intervals, set, judge, signed thresholds, limits, monitoring. Use at go-live, a new version or an audit.

  • model-selection

    Choose a model for an AI use case on your own eval suite (cost per success, latency, hosting) and write the decision record. Use for "which model?", a new release or a cost review.

  • multi-turn-evals

    Evaluate whole conversations. Use when a chat assistant forgets, contradicts itself or drifts over turns while one-shot tests pass, or a per-turn average or simulated-chat score is offered as ready.

  • process-ai-redesign

    Redesign a business process with AI: autonomy level and gate per step, three scenarios with ROI lines, trade-offs, migration path. Use for "automate this process" or an efficiency target.

  • production-evals-monitoring

    Watch an AI feature's answer quality in production (samples, judges, alerts, drift). Use before go-live, for a post-market monitoring plan, or when a pass rate drops and someone wants to roll back.

  • prototype-handover

    Decide what a PM-built AI prototype becomes: rebuilt by engineers, registered at its EUC tier, or retired. Use when a prototype gets users, nears its kill date, or someone asks to keep it.

  • red-team-evals

    Plan or read attack tests on an AI assistant or RAG feature: what users can make it say or disclose. Use for jailbreak results, a vendor resistance figure, "robust" wording, a campaign before release.

  • release-to-production

    Plan the release of an AI change (prompt, model, KB) through offline eval, shadow, canary and A/B if the sample allows. Use for "can we ship this?" or a provider model update.

  • retrieval-evaluation

    Evaluate a RAG or search feature in two stages, retrieval then generation. Use when answers are wrong or cite the wrong document, before changing the LLM, or for a new embedding model or re-index.

  • synthetic-eval-data

    Generate synthetic eval inputs when real cases are scarce, without fooling yourself. Use pre-launch, for a rare risky class, when privacy blocks real data, or before "generate 1,000 test questions".

  • system-prompt-and-guardrails

    Write or tune an LLM feature''s system prompt and guardrails. Use for a first prompt, a jailbreak, injection or leak, a guardrail refusing too much (false positives), or a redteam result to judge.

  • trace-review-tool

    Spec the thin trace-labelling app a coding agent builds for domain experts, and the fields each model call must log. Use for "build us a labelling tool" or before experts review outputs.

View source on GitHub

More from alexandre-darmon/ai-pm-skills

All 30 plugins →

Design an agent's loop, tools, permissions, stop conditions and traces from its autonomy levels, before any framework. Use for "make it an agent", a new tool or permission, or a framework debate.

29 skills

1

agent-evals

claudecodex

Test an agent that calls tools or acts on systems and gate its autonomy on measured reliability. Use for "how do we test our agent?", a level change, a new tool, or "it says done, nothing changed".

29 skills

1

Classify an AI use case under the EU AI Act (role, risk category, obligations, questions for counsel). Use for a new AI use case, an autonomy level change, or a vendor's AI Act claim.

29 skills

1

ai-adoption-plan

claudecodex

Plan and prove adoption of an AI tool on outcomes: correct results per user, overrides, over-trust test, training by role and level, art. 4 evidence. Use when go-live nears or use stalls.

29 skills

1

Specify what an AI feature reads and writes. Use on "what data do we need", a DPO asking what is processed, or to set up or review a RAG store: stale answers, mixed sensitivity, unowned docs.

29 skills

1

Use when someone wants to prototype or spike an AI feature or agent on the real model before a PRD, or has spike results to read, write up as a go note or offer as validation evidence.

29 skills

1

Maintain production-evals-monitoring?

This page updates itself from your repo. Add the badge to your README, or get it Featured to pin it above the organic results with a followed link to your site.

claude & codex plugins30 plugins