Prove an LLM/RAG/agent feature is correct with numbers a release can gate on — author eval suites in DeepEval with the metric definitions baked in: faithfulness and answer relevancy (generator), contextual precision/recall/relevancy (retriever), hallucination against ground truth, tool-call correctness for agents, plus custom G-Eval criteria, bias, and toxicity. Builds a versioned golden dataset FIRST, picks the metric that matches the actual failure mode (RAG hallucination -> faithfulness; retrieval miss -> contextual recall; wrong tool -> tool correctness), and turns every threshold into a pytest / `deepeval test run` exit code so a regression blocks the pipeline. Enforces guardrails against unpinned LLM-as-judge models, exact-string-matching non-deterministic output, contaminated goldens that leak the prompt's own examples, single-run scores with no variance, averaging unrelated metrics into one number that hides a failing dimension, and getting metric direction wrong (since deepeval 4.2.0 hallucination/bias/toxicity score the clean share and pass at score >= threshold; older versions inverted that, so the version is pinned and thresholds re-baselined across the change). Adds an adversarial/safety slice (prompt injection, jailbreak, PII) and judge calibration against human labels.
Plugin llm-eval from marketplace qa-ai-repo in pnakhat/qa-ai-repo. Commands read from the repo's manifest — 1 month ago.
llm-evalAuthor LLM/RAG/agent evaluation suites in DeepEval that prove a feature is correct with gated numbers, not vibes. Use when asked to "eval an LLM", "test a prompt", "measure RAG quality", "check for hallucination", "score answer relevancy", "verify tool calls", or gate a release on model output quali
llm-eval-authorUse to design and build LLM/RAG/agent evaluation suites in DeepEval that gate a release on output quality. It elicits or derives the golden dataset and the failure mode to guard against, picks the metrics that match it (faithfulness/answer-relevancy for the generator, contextual precision/recall for
Author and maintain resilient Playwright end-to-end tests using the Page Object Model, fixtures, stable user-facing locators, and web-first assertions. Enforces guardrails against flakiness, brittle selectors, and shared state. Drives Playwright's built-in planner, generator, and healer agents (playwright-test MCP) and refactors their output to project conventions; also includes the live-browser Playwright MCP server.
1 skill · 1 agent · 2 MCP servers
Test web UIs for accessibility to WCAG 2.2 AA using both automation and the manual checks automation can't perform. Wires axe-core (`@axe-core/playwright`, `jest-axe`, `cypress-axe`) in as a fixture- and component-level assertion, then adds the human review that catches the rest: keyboard-only navigation, focus management and order, screen-reader semantics (roles, names, states), and meaningful accessible names. Automated tooling reliably catches only ~30–40% of real accessibility issues, so an axe pass is a floor, never a certificate. Enforces guardrails against `aria-label` spam, div-soup with role hacks, disabling axe rules to go green, and treating a clean axe run as full accessibility coverage. See the skill's `reference.md` for working setup code, rule/tag config, contrast thresholds, and a copy-pasteable manual checklist.
1 skill · 1 agent · 1 MCP server
Catch breaking API changes before they ship — consumer-driven contracts (Pact) and spec-first validation (OpenAPI/Spectral/Schemathesis/oasdiff), across REST, GraphQL, and event/message boundaries. Enforces guardrails that reject exact-value matching in favour of type/shape matchers, require every pact and spec to carry a version + git sha, replace shared fixtures with per-interaction provider states, and treat any backward-incompatible change as a versioned breaking change. Wires provider verification, breaking-change diffs, and blocking can-i-deploy / record-deployment gates into CI so an incompatible release is stopped, not merely reported.
1 skill · 1 agent
Detect and quantify flaky tests — compute a suite-wide flake rate and a per-test flake score from reruns and CI history, and flag any test that passes-on-retry or fails only in certain orders. Classify each flake by root cause (async/timing, shared state & ordering, external dependencies, animations, time/locale/randomness, resource leaks) using tell-tale signals, then fix it at the root: web-first assertions over sleeps, per-test isolation over shared fixtures, mock only what you don't own, frozen clocks and seeded RNG, disabled animations, and disposed resources. When a fix can't land immediately, quarantine with discipline — tag `@flaky`, move it to a non-blocking lane, and require an owner, a tracking issue, and an SLA — never blanket retries. Enforces guardrails against hiding flake behind `retries`, deleting the failing test to make CI green, and quarantining forever with no owner or SLA; a quarantine un-quarantines only after N consecutive green runs, and a breached SLA auto-escalates to fix-or-delete.
1 skill · 1 agent
Measure how good your unit tests actually are — set up Jest code coverage, then Stryker mutation testing to prove the tests catch real bugs (not just execute lines). Interpret mutation score, kill survived mutants, and gate CI on both. Enforces guardrails against coverage theater: no assertion-free tests, no gaming the score by excluding mutators or files, every fix must kill a specific survived mutant, mutation is scoped to high-risk logic (changed-code on PRs, full run nightly), and CI is gated on a Stryker break threshold — not coverage % alone.
1 skill · 1 agent
Prove a system meets its speed and scale targets under realistic load — backend load, stress, soak, and spike testing with k6, and frontend web-vitals with Lighthouse. Defines SLOs (p95/p99 latency, error rate, throughput/RPS, and LCP/INP/CLS) FIRST, models a workload from real traffic with think time and ramp stages, then expresses every threshold as a pass/fail CI gate so a regression blocks the pipeline instead of being noticed in production. Always reports percentiles, never averages, because an average hides the slow tail that users actually feel. Enforces guardrails against averages-over-percentiles, zero-think-time hammering, testing an under-provisioned or localhost environment and extrapolating to production, drawing a trend from a single run, and chasing latency while ignoring the error rate at load.
1 skill · 1 agent
Collection of example skills demonstrating various capabilities including skill creation, MCP building, visual design, algorithmic art, internal communications, web testing, artifact building, Slack GIFs, and theme styling
19 skills
Comprehensive PR review agents specializing in comments, tests, error handling, type design, code quality, and code simplification
1 command · 6 agents
Build, preview, and test UI components with Storybook.
7 skills
Agentic QE (external: maintained in proffesor-for-testing/agentic-qe, pinned to v3.15.17): a quality-engineering fleet with 11 QE agents, 9 skills, 9 commands, the agentic-qe MCP server pinned to the plugin version, and the aqe-mod guard that refuses destructive operations on AQE learning data
9 skills · 9 commands · 11 agents · 1 MCP server
Automates browser interactions for web testing, form filling, screenshots, and data extraction
12 skills
Install the "AAS QA & Test Automation" workflow plugin from Agentic Awesome Skills.
14 skills
This page updates itself from your repo. Add the badge to your README, or get it Featured to pin it above the organic results with a followed link to your site.