Eval-gated LLM fine-tuning lifecycle (LoRA/QLoRA, DPO, GRPO/RLVR, vision SFT, quantized export). Use when fine-tuning an open-weights model end-to-end: building the eval harness first, selecting method and model, preparing data, training with Unsloth or TRL, gating checkpoints, and exporting GGUF/FP8.
Use when paired or branched ML experiments need uncertainty estimates, grouped splits, selection-bias controls, power analysis, or abstention calibration.
LLM and prompt evaluation with Promptfoo, DeepEval and Ragas, Giskard vulnerability scans, Langfuse tracing, and golden-dataset regression suites. Audits eval suites for methodology errors. Agents (prompt-eval-reviewer, llm-red-team-planner) review an eval suite and plan adversarial probe campaigns across the OWASP LLM Top 10.
ML model testing: Giskard and Deepchecks validation, Evidently drift monitoring, Fairlearn fairness metrics, and model performance regression gates. Assigns risk tiers with the fairness and explainability evidence each tier requires, and wires Jupyter notebook CI with papermill, nbval, testbook and nbstripout.
AI-assisted test generation: LLM-written tests from natural-language specs with confidence scoring, spec coverage gap mapping, and model-based testing. Builds state-machine models and turns them into test suites, and adds adversarial review agents (ai-test-curator, ai-test-shallow-coverage-critic) that catch hallucinated APIs, weak assertions, and happy-path-only coverage.