llm-observability

Plugin llm-observability from marketplace contextjet in ContextJet-ai/awesome-llm-observability. Commands read from the repo's manifest — 2 days ago.

Skills (26)

• add-llm-evals

  Use this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Cov

• add-llm-fallbacks

  Use this to make an LLM app resilient to provider failures, rate limits, timeouts, and outages. Trigger on "handle LLM API errors", "add retries/fallbacks", "the app breaks when OpenAI is down", "rate limit errors", "make my LLM calls reliable", "timeout handling". Add retries, timeouts, and model/p

• add-llm-guardrails

  Use this to add safety and security guardrails to an LLM/agent app - blocking prompt injection, PII leakage, jailbreaks, toxic output, off-topic responses, or invalid structured output. Trigger on "add guardrails", "prevent prompt injection", "stop PII leaks", "validate the model's output", "make th

• annotate-traces-for-review

  Use this to set up human review and annotation of LLM traces, so people (often domain experts) can label outputs, do error analysis, and build a trustworthy golden dataset. Trigger on "review my LLM outputs", "have an expert label these", "error analysis", "annotate traces", "build a golden dataset"

• build-eval-dataset

  Use this to build a good evaluation dataset for an LLM app, the part everyone underestimates. Trigger on "make an eval set", "what should I test my LLM on", "I don't have test data for my prompt", "build a golden dataset", or before setting up evals. A great eval set beats a great metric; garbage-in

• check-answer-consistency

  Use this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement. Trigger on "is this answer reliable", "flag low-confidence answers", "cheap hallucination check", "confidence score without a ground truth", "self-cons

• choose-observability-stack

  Use this to recommend an LLM observability / evaluation tool or stack for a specific situation. Trigger on "which observability tool should I use", "compare Langfuse vs Phoenix vs LangSmith", "what's the best LLM monitoring for us", or picking an eval/tracing/gateway tool given constraints (self-hos

• collect-user-feedback

  Use this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedba

• compare-llm-models

  Use this to pick or switch the LLM behind a feature, based on evidence instead of hype or the newest release. Trigger on "which model should I use", "is GPT/Claude/Gemini/Llama better for this", "should I switch models", "can a cheaper model do this", "compare models for my use case". Evaluate on YO

• debug-agent-from-traces

  Use this to diagnose WHY an LLM agent or chain produced a wrong, empty, slow, or expensive result, by reading its observability trace. Trigger on "my agent gave the wrong answer", "the chain returned nothing", "why is this so slow/expensive", "debug this trace/run", or when a trace tree is available

• detect-hallucinations

  Use this to detect when an LLM is making things up, so you can flag or block confident-but-wrong answers before users see them. Trigger on "detect hallucinations", "is the model making this up", "flag unreliable answers", "hallucination check", "confidence scoring for LLM output", or hardening a RAG

• estimate-llm-cost

  Use this to estimate what an LLM call or feature will cost, and to compare models on price, before or after shipping. Trigger on "how much will this cost", "estimate my OpenAI/Anthropic bill", "is a cheaper model worth it", "cost of this prompt", "project my LLM spend". Ships a runnable, tested calc

• eval-driven-development

  Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate on my agent", or any p

• instrument-llm-observability

  Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer ven

• measure-agent-task-success

  Use this to measure whether an AI agent actually completed its task end to end, not just whether individual LLM calls looked fine. Trigger on "is my agent working", "measure agent success rate", "evaluate my agent", "how good is my agent", "agent completion rate", or evaluating a multi-step/tool-usi

• monitor-rag-quality

  Use this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG", "improve RAG accuracy

• optimize-prompts

  Use this to improve a prompt systematically instead of hand-tweaking it by feel. Trigger on "optimize my prompt", "make this prompt better", "the prompt isn't working well", "auto-tune my prompt", "few-shot example selection", or when prompt quality has plateaued. Optimize against an eval set with a

• red-team-llm-app

  Use this to adversarially test an LLM/agent app before attackers do - prompt injection, jailbreaks, data exfiltration, tool misuse, and unsafe output. Trigger on "red team my LLM", "test for prompt injection", "is my agent secure", "jailbreak testing", "security review of my AI app", especially befo

• redact-pii-for-tracing

  Use this when adding LLM observability to an app that handles sensitive data (finance, healthcare, PII) and you must NOT ship raw prompts/PII to a third-party tracing backend. Trigger on "redact traces", "PII in observability", "can we self-host tracing for compliance", "GDPR/HIPAA/SOC2 and LLM logg

• reduce-llm-cost

  Use this to cut the cost of an LLM app using observability data. Trigger on "my OpenAI/Anthropic bill is too high", "reduce token usage", "the app is expensive", "optimize LLM cost", "why am I spending so much on the API". Find the expensive spans first (measure), then apply the cheapest wins. Don't

• scrub-pii-from-text

  Use this to strip PII (emails, credit cards, SSNs, phone numbers, IPs, IBANs) out of text before it is logged to a tracing backend or sent to a third-party model. Trigger on "redact PII", "scrub sensitive data", "mask PII before logging", "don't send customer data to the tracing tool", especially fo

• set-up-ab-testing

  Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things for real users". Meas

• set-up-drift-alerts

  Use this to catch an LLM app silently getting worse in production - quality dropping, cost creeping up, inputs shifting away from what you tested. Trigger on "monitor my LLM in production", "alert me when quality drops", "detect drift", "my app got worse and I didn't notice", "set up monitoring/aler

• trace-based-testing

  Use this to turn real production traces into a regression test suite, so bugs you saw once never come back. Trigger on "turn traces into tests", "capture this bug as a test", "build tests from production data", "my eval set is out of date", or after debugging a production incident. This is the loop

• trace-multi-agent-system

  Use this to add observability to a multi-agent or agentic system (multiple agents, sub-agents, tool loops, handoffs). Trigger on "trace my agents", "my agent crew is a black box", "which agent failed", "debug my LangGraph/CrewAI/AutoGen/agent workflow", "the agents loop forever". Get a clear span tr

• validate-genai-spans

  Use this to check that your LLM tracing actually emits complete, spec-compliant spans, so cost/latency/model dashboards downstream are not full of holes. Trigger on "are my traces complete", "validate my instrumentation", "my spans are missing fields", "lint my OTel GenAI spans", "test my tracing".

Similar plugins