batch-inference

Plugin batch-inference in danielrosehill/Batch-Inference. Commands read from the repo's manifest — 5 days ago.

Skills (16)

• anthropic-batch

  Run a bulk job through the Anthropic Message Batches API — Claude Haiku 5.5 by default, half price, usually done in minutes — from a JSONL file of prompts, with structured JSON output, a cached system prompt, resume-by-batch-id and cost logging. The default lane for batch work. Use when the user say

• batch-defaults

  Pick the model, provider and lane for any bulk or batch LLM job (labelling, classification, extraction, synthetic data, summarising hundreds or thousands of items). Defaults: Claude Haiku 5.5 on the Anthropic Batches API for judgement work, DeepSeek V4.1 Flash for cheap volume (batch or parallel syn

• batch-research-check

  Read the research notebook before choosing a provider, model or lane for a bulk/batch LLM job, or before quoting a batch price — per-task screening tables (which model was picked for a task type and why), dated verdicts (use / avoid / watch), the latest price per provider and model, provider gotchas

• batch-research-record

  Write what a batch/bulk LLM job taught us into the research notebook — the run record with measured tokens and cost, model screening results in the per-task tables, new or changed provider prices with the date checked, a use/avoid/watch verdict, provider gotchas, and a sample of the outputs. Use at

• cheapest-batch

  Run a bulk LLM job as cheaply as possible — DeepSeek V4.1 Flash via the OpenRouter Batch API or DeepSeek's own API off-peak, whichever costs less today, with reasoning off and JSON-mode fallbacks. Use when the user says "cheapest batch possible", "as cheap as possible", "use DeepSeek", "doesn't need

• check-api-prices

  Check today's LLM API prices across every lane in use: OpenRouter sync vs batch (live API), what each `~vendor/*-latest` alias really serves, Anthropic list and Batch prices, and DeepSeek direct peak and off-peak. Then record them, dated, in the batch notebook. Use when the user asks "what does X co

• find-batch-run

  Look up past batch runs: which model, what it cost, how it did. Use on "what did I use for that", "did this work last time".

• list-openrouter-models

  Read OpenRouter's live model list for a bulk job. Lists every batch-capable model with batch vs sync price and which is cheaper, finds sync instruct models fit for bulk work (reasoning can be off, JSON mode, cheap), shows who hosts a model, and live-probes a shortlist. Use for "which models support

• local-bulk-run

  Run a bulk LLM job on local or self-hosted inference (Ollama, llama.cpp, vLLM, LM Studio or the user's own servers), from the same JSONL as the cloud runners. Checks the backend is up, estimates hours against the cloud price, pilots, then runs resumably. Use for "run it locally", "use Ollama", "use

• log-batch-run

  Add a finished batch job to the run ledger: data size, model, cost, rating, notes. Use on "log this run", "add a run record".

• onboarding

  First-time setup for batch inference. Finds local model servers, records the user's backends (Ollama, llama.cpp, vLLM, LM Studio, their own servers, cloud APIs), and takes their guide to when each should be used, plus spend rules, secrets wrapper and notebook location. Rerun to change any of it. Use

• openrouter-batch

  Run a bulk job through the OpenRouter Batch API (POST /api/v1/batches) for any non-Anthropic model — DeepSeek, GPT, Gemini, Qwen, GLM, Mistral — from a JSONL file of prompts, after finding which models are batch-capable and checking whether batch is actually cheaper than sync today. Use when the use

• pick-batch-model

  Name the specific model (exact OpenRouter or Anthropic id) for a bulk/batch job by use case — labelling, extraction, LLM-as-judge, summarising, translation, distillation teacher, synthetic-data seeds, synthetic-data writing, long-form prose — from measured price, a measured diversity probe and EQ-Be

• setup-batch-eval

  Set up an evaluation for a batch inference job, to choose or check its model before the full run. Installs the companion ai-eval plugin if it's missing, then gives it the job's real rows, prompt, schema, volume and budget. Use for "set up an eval for this batch job", "evaluate models for this batch"

• sync-bulk-run

  Run a bulk LLM job as parallel sync calls instead of a batch, usually on a cheap instruct model such as DeepSeek V4.1 Flash. Finds and probes candidates from OpenRouter's model list, compares cost against batch, pilots, then runs resumably with a hard spend cap and real measured cost. Use when sync

• synthetic-data-generation

  Generate varied, non-repetitive synthetic data in bulk (training pairs, test inputs, personas, scenarios, fake dictations/emails/messages) on batch APIs. Covers the seed-then-write recipe, which models to rotate, and the dedupe and diversity checks, based on the user's 11-model diversity probe (2026

Similar plugins