mechanist

Plugin mechanist from marketplace mechanist in zjunlp/Mechanist. Commands read from the repo's manifest — 1 month ago.

Skills (85)

• ablation-planner

  Use when main results pass result-to-claim (claim_supported=yes or partial) and ablation studies are needed for paper submission. The external LLM reviewer (via llm-chat MCP) designs ablations from a reviewer's perspective, CC reviews feasibility and implements.

• analyze-results

  Analyze ML experiment results, compute statistics, generate comparison tables and insights. Use when user says "analyze results", "compare", or needs to interpret experimental data.

• arxiv

  Search, download, and summarize academic papers from arXiv. Use when user says "search arxiv", "download paper", "fetch arxiv", "arxiv search", "get paper pdf", or wants to find and save papers from arXiv to the local paper library.

• auto-claim

  Workflow 1: Claim-stage pipeline, controlled by two orthogonal axes. BEHAVIOR_SOURCE selects the behavior stage: `given` (default; behavior taken from task.md and assumed to hold — no ideation, no novelty, no M0), `given-validation` (behavior taken from task.md but the experiment plan opens with an

• auto-experiment

  Workflow 1.5: Bridge between idea discovery and auto review. Reads EXPERIMENT_PLAN.md, routes mechanism family inline (Phase 1.5), implements experiment code, deploys to GPU, and collects initial results. Use when user says \"implement experiments\", \"experiment\", \"deploy the plan\", or has an ex

• auto-iteration-loop

  Autonomous research review loop that consumes /auto-verify's four-state output (PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS / deferred) and routes each claim to the right back-edge — brief audit, two-phase FAIL handling (variant-integrity fix then optional claim-stage re-entry), main-experim

• auto-verify

  Workflow 1.75: stress-test claims (regardless of main-experiment verdict) by swapping method, dataset, and model, then judging whether each variant agrees with the main experiment. Three stages with two integrity gates: Stage 1 audits the main experiment's eval method for every target claim; Stage 2

• verify-pick-alternatives

  Sub-skill of /auto-verify. Given a claim, choose one method swap, one dataset swap, and one model swap that most strongly stress-test the claim. Harvests candidates from existing research; calls /research-lit only when coverage is thin. Use when user says \"pick swaps for claim\", \"choose alternati

• auto

  Autonomous pipeline: claim → experiment (mechanism routing folded in) → verify → iteration. Each stage is delegated to an isolated agent with its own context window and configurable model. Gates are AUTO_PROCEED-governed; defaults run end-to-end without human input. Use when user says \"auto pipelin

• data-rule

  The single place for every data constraint an experiment must satisfy — dataset provenance (existing → adapted → constructed), clear train / validation / test splits, labels that reflect the target behavior, and the minimum data amount. Use whenever an experiment chooses, adapts, or constructs a dat

• experiment-audit

  Audit the experimental **methodology** integrity for a specific claim (Checks A–F: GT provenance, score normalization, result-file existence, dead code, scope, eval-type). Uses cross-model review (external LLM reviewer via llm-chat MCP). The output `overall_verdict` (PASS/WARN/FAIL) is THIS claim's

• experiment-plan

  Turn a refined research proposal or method idea into a detailed, claim-driven experiment roadmap. Use after `research-refine`, or when the user asks for a detailed experiment plan, ablation matrix, evaluation protocol, run order, compute budget, or paper-ready validation that supports the core probl

• experiment-queue

  SSH job queue for multi-seed / multi-config ML experiments with OOM-aware retry, stale-screen cleanup, wave-transition race prevention, and phase-dependency enforcement. Use when user says "batch experiments", "queue experiments", "run grid", "multi-seed sweep", "auto-chain experiments", or when `/r

• experiment-tips

  Routing entry point for experiment-protocol tips that prevent silent reproducibility / overclaim failures. Use when EXPERIMENT_PLAN.md is about to become runnable code and any of these is in scope: ImageNet / torchvision preprocessing, steering coefficient (α / dose / magnitude), steering block / la

• finetune-hyperparameter-sweep

  Fine-tuning LR protocol — full FT, LoRA / QLoRA / DoRA / PEFT adapter, across SFT, DPO, and GRPO / PPO / RL objectives. Fires on ANY fine-tune, including (especially) when the plan already fixes a learning rate or copies one from a reference paper: a fixed LR is one grid point, never the answer, so

• image

  Canonical ImageNet eval preprocessing — square 256×256 resize → 224 center crop → ImageNet mean/std — for CV experiments probing ImageNet-pretrained backbones (ResNet, ViT, VGG, EfficientNet). Use this skill whenever a `torchvision.transforms` / `PIL` pipeline is being written, audited, or debugged

• multiple-choice-evaluation

  How to grade any multiple-choice / A-B / A-D letter task when the score is read out of the model''s free-form output — any eval that maps a generation to a choice letter, whatever the domain. Use whenever the plan parses the letter with a regex like `[A-D]` / `re.search(r"[AB]", ...)`. Covers why na

• steering-block-selection

  How to choose where (and how many sites) to intervene for any operation on internal representations — activation patching, steering, CAA, DAS, SAE feature scaling, attribution patching. Use whenever the plan declares a target block / layer / site, especially when it hard-codes a single index (`targe

• steering-coefficient-tuning

  How to set the strength of any additive intervention on internal representations — steering vectors, CAA, DAS dose-response, representation engineering, SAE feature scaling, ROME-style edits. Use whenever the plan pins a steering strength (`α`, `β`, `dose`, `magnitude`, `scale`, `coefficient`, `k`)

• hypothesis-batch

  Automated pipeline for generating and refining multiple research hypotheses.

• idea-creator

  Generate and rank research ideas given a broad direction. Use when user says "brainstorm ideas", "generate research ideas", "what can we work on", or wants to explore a research area for publishable directions.

• impact-check

  Assess whether the research problem/behavior is important — its potential value and reach. Use when user says "impact check", "check impact", or wants to judge whether an idea/behavior matters before committing.

• mechanic-db-search

  Paper retrieval via the cloud SEARCH service. The Agent builds a decomposed query JSON from its task context (preferred) or submits a polished free-form English query; the cloud service performs multi-ranker retrieval and fusion. Use as one of an important paper sources.

• mechanism-audit

  Audit the **mechanistic experiment rigor** for a specific claim. Catalogue currently has six slots A–F: A (steering coefficient sweep) is implemented; B–F are reserved for future checks (direction extraction quality, site/layer selection, n_effective sufficiency, probe-vs-causal disentanglement, int

• mechanism-behavior-discovery

  Mine behavioral regularities in neural-network (LLM / multimodal) models — the upstream half of the project''s mission (find a behavior worth explaining, then investigate the mechanism behind it). Use this skill when the task is open-ended: surface a *new* behavioral phenomenon — a candidate claim /

• mechanism-explore

  Macro-level strategic directions for investigating the *mechanism* behind a model behavior — the downstream half of the project''s mission (mine LLM behaviors, then explain the mechanism behind them). Use once a phenomenon is observed in a model — whether already established/known or freshly mined b

• SHAP

  SHAP (SHapley Additive exPlanations) is a unified, game-theoretic framework for local feature attribution. Treating a model's input features as players in a coalition game, SHAP assigns each feature its Shapley value — the average marginal contribution of that feature to the prediction across all po

• fastshap

  Use this skill when you need to train amortized Shapley value explainers using FastSHAP, generate real-time local feature importance explanations for machine learning models (tabular or image), train surrogate models for feature masking, or understand how FastSHAP's KernelSHAP-inspired training obje

• shap

  Use this skill when working with SHAP (SHapley Additive exPlanations) to explain machine learning model predictions, compute feature importance, generate SHAP values for tree ensembles (XGBoost, LightGBM, CatBoost, scikit-learn), deep learning models (TensorFlow, Keras, PyTorch), NLP transformers, o

• mechanism-skills

  Routing entry point for eleven families of mechanistic-interpretability methods that localize *which* internal object (layer, attention head, neuron, SAE feature, weight, or input feature) drives a model''s behavior, *how influential* it is, and *what changes* when it is intervened on. Use this skil

• causal-attribution

  Causal Attribution methods constitute the gold standard for localization in Mechanism Interpretability. Unlike correlation-based analyses, these techniques identify which internal objects are causally responsible for a specific model behavior by systematically measuring the effect of controlled inte

• dissecting-factual-predictions

  Analyze and dissect factual recall in auto-regressive language models using attention knockout, hidden state analysis, and intervention techniques on GPT-2 and GPT-J models

• attribution-patching

  Use when analyzing neural network circuits, performing attribution patching, automated circuit discovery, or investigating model interpretability through edge attribution methods in transformer models

• rome-model-editing

  Use this skill when you need to edit factual knowledge in large language models like GPT-2 or GPT-J, perform causal tracing to understand model behavior, or implement Rank-One Model Editing (ROME) to modify specific factual associations without retraining

• Circuit

  Circuit discovery methods localize the minimal subgraph of a transformer's computational graph (a set of nodes such as attention heads and MLPs, together with the edges that connect them) that is causally responsible for a specific model behavior. Rather than scoring individual objects in isolation,

• eap-ig

  Use this skill to discover and analyze computational circuits within autoregressive transformer language models via attribution methods like Edge Attribution Patching with Integrated Gradients (EAP-IG), enabling mechanistic interpretability studies and circuit evaluation.

• automatic-circuit-discovery

  Activates when working on mechanistic interpretability of transformers, specifically for automated circuit discovery in transformer models using computational graph editing and analysis.

• saelens

  Use this skill when working with Sparse Autoencoders (SAEs) for mechanistic interpretability of language models, including training SAEs, loading pre-trained SAEs, analyzing neural network features, or integrating SAEs with TransformerLens, HuggingFace Transformers, or other PyTorch-based models.

• Feature Dictionary Learning

  Feature Dictionary Learning methods address the polysemanticity of neuron-level units by decomposing a dense internal activation (e.g. a residual-stream state or MLP output) into a sparse weighted sum of directions drawn from a large over-complete dictionary. The dictionary contains far more directi

• crosscoder-learning

  Use this skill when working with sparse autoencoders (SAEs), crosscoders, dictionary learning on neural network activations, training SAEs/crosscoders from scratch, loading pretrained dictionaries, caching model activations, or comparing model internals across fine-tuned model pairs using the dictio

• ica-lens

  Use this skill when applying Independent Component Analysis as a training-free interpretability lens — decomposing a target activation site (residual stream, MLP output, attention-head output, or any cached hook point) into maximally non-Gaussian directions and treating each direction as a candidate

• transcoder-circuits

  Use this skill when working with transcoder-based circuit analysis of large language models, including training transcoders, analyzing MLP sublayers, reverse-engineering LLM circuits, and creating feature dashboards for interpretability research.

• gradient-detection

  Gradient Detection methods localize influential internal objects by scoring them with the sensitivity of a scalar target $F(x)$ (e.g., a logit, margin, or loss) with respect to an object $o_j$: $s_j(x) = \phi(\nabla_{o_j} F(x), o_j)$, where common instantiations include the gradient norm $s_j = \|\n

• inputs-and-layer-wise-states

  Analyze and visualize layer-wise gradient behaviors in LLMs during fine-tuning for fast vs slow thinking tasks, calculate gradient statistics, and understand training patterns across different model layers

• intermediate-outputs

  Use this skill when working with circuit discovery in language models, mechanistic interpretability, activation patching, attribution patching, or Layer-wise Relevance Propagation (LRP) for neural network analysis

• linguistic-regions-llm

  Use this skill when working with linguistic region analysis in Large Language Models, including data preprocessing for multilingual training, region-based model training with DeepSpeed, and extracting/visualizing linguistic regions in transformer models

• Magnitude Analysis

  Magnitude Analysis methods serve as a fundamental heuristic in interpretability, operating on the premise that internal elements with larger numerical values often exert greater influence on the model’s computation. It scores internal objects via a scalar function to identify salient components

• dynamic-components

  Identify and manipulate language-specific neurons in multilingual Large Language Models (LLMs) to understand and control language-specific behaviors in models like LLaMA-2, BLOOM, OPT, Mistral, and Phi-2

• layer-wise-representation

  Use this skill when you need to enhance the truthfulness of Large Language Models (LLMs) or reduce hallucinations in model outputs. This skill provides TruthX, an inference-time method that edits LLM internal representations to control truthfulness and mitigate hallucinations.

• static-parameters

  Analyze and manipulate massive values in LLM attention mechanisms, particularly for understanding contextual knowledge processing in transformer models with RoPE

• Multi-Modal Interpretability

  Multi-Modal Interpretability methods explain *vision* and *vision-language* models by linking each internal unit (a convolutional neuron, attention head, residual-stream channel, or SAE feature) to a *natural-language concept* drawn from an external concept set $\mathcal{C}$. The unifying constructi

• clip-dissect

  Use this skill when you need to automatically describe or interpret the functionality of individual neurons in deep neural networks (DNNs) using CLIP-based semantic analysis, perform mechanistic interpretability research on vision models, dissect convolutional or transformer-based image classifiers,

• zennit-crp

  Use this skill when working with Concept Relevance Propagation (CRP) and Relevance Maximization for explainable AI in PyTorch models, including generating concept-conditional heatmaps, feature visualizations, attribution graphs, and identifying which latent concepts neural networks use for predictio

• Neural Feature Learning

  Neural Feature Learning methods open the black box of *how* a trained neural network discovers useful representations from data, by deriving the directions along which each layer's effective input geometry is reshaped during training. The unifying object is the Neural Feature Matrix (NFM) — for a fu

• deep-neural-feature-ansatz

  Use this skill when working with the Deep Neural Feature Ansatz (DNFA) — verifying feature learning in neural networks, training fully connected networks on image/tabular datasets, computing Neural Tangent Kernels (NTK) and Neural Network Gaussian Processes (NNGP), or reproducing experiments from th

• recursive-feature-machines

  Use this skill when working with Recursive Feature Machines (RFM) for kernel-based machine learning, feature learning, or when implementing backpropagation-free models that learn features similarly to neural networks.

• tp4-feature-learning

  Use this skill when working with infinite-width neural networks for feature learning, replicating Word2Vec or MAML experiments from the Tensor Programs series (TP4), or implementing infinite-width limits (GP, NTK, muP) for meta-learning and word embedding tasks.

• convrfm

  Use this skill when working with convolutional neural feature ansatz (CNFA), deep convolutional recursive feature machines (Deep ConvRFM), or tasks involving kernel-based feature learning with convolutional architectures, VGG networks, patch-based Jacobian computation, and EGOP (expected gradient ou

• Probing

  Probing methods interpret model signals by training an auxiliary predictor (often linear) to decode a labeled property y from an internal vector $z$ (e.g., the residual stream state $x_l$ at layer $l$). Operationally, probing treats the model as a frozen feature extractor and assesses decodability:

• residual-stream-states

  Use this skill when working with LLM hallucination detection, probing internal representations of language models, analyzing model correctness, or conducting intrinsic evaluation experiments on models like Mistral and Llama-3

• sae-feature-activation-state

  Use this skill when working with Sparse Autoencoders (SAEs) for feature analysis, particularly for studying feature splitting, absorption, and attribution in language models. Activate for tasks involving SAE feature ablation, probing experiments, or analyzing how SAE latents affect model outputs in

• Representation and Parameter Analysis

  Representation and Parameter Analysis interprets and controls a model by directly manipulating its two kinds of internal objects — features (the hidden-state activations produced during a forward pass) and weights (the parameters of the target model). The analysis side relies on standard linear-alge

• task-vectors

  Use this skill when working with task arithmetic for editing neural network models, including creating task vectors from pre-trained and fine-tuned checkpoints, combining them via arithmetic operations (negation, addition, analogies), and applying them to CLIP vision models for multi-task learning.

• representation-engineering

  Use this skill when working with Representation Engineering (RepE) for AI transparency, monitoring, or controlling internal representations of large language models including truthfulness detection, emotion control, harmlessness steering, and memorization analysis.

• steer-features

  Use this skill for feature-level steering of models — locating the internal feature that drives a target behavior, scoring and selecting it by its effect on the model's output, and directly amplifying or shrinking that feature's activation during generation to control behavior. Applies to features r

• steering-vectors

  Work with Llama 2 model steering using Contrastive Activation Addition (CAA) for behavioral control, including generating steering vectors, evaluating model behavior, and analyzing activation patterns

• Vocabulary Projection

  Vocabulary Projection methods interpret internal model states by projecting them through the unembedding matrix to obtain a distribution over the vocabulary. The core idea is that the unembedding matrix, which maps the final hidden state to output logits, can serve as a universal decoder for interme

• attention-head-output

  Use this skill when you need to analyze and interpret the internal workings of large language models layer by layer, visualize hidden states and predictions across transformer layers, or understand how models like Llama-3.1-8B and Qwen-2.5-7B make predictions at each layer using the Logit Lens techn

• neuron-value-weight

  Analyze transformer feed-forward layers as key-value memories, extract activations, identify trigger examples, and compute key-value agreement in transformer language models

• residual-stream-state

  Use this skill when working with transformer model interpretability, analyzing layer-by-layer predictions, training tuned lenses to understand intermediate representations, or peeking into iterative computations of transformers

• mguide

  Mechanist's single conversational front door, with three parallel tracks: (A) when the user wants to run an experiment, work out the research requirements through conversation, settle the two parameter axes of /auto (behavior-source × mechanism), write a compliant task.md into the project root on th

• mhistory

  Generate a structured, publication-quality research-history markdown article for a given topic. Uses the cloud `mechanic_database` SEARCH service via skill `/mechanic-db-search` (TWO PARALLEL passes — `temporal_mode=history` for the long arc + `temporal_mode=recent` for the frontier), then supplemen

• monitor-experiment

  Monitor running experiments, check progress, collect results. Use when user says "check results", "is it done", "monitor", or wants experiment output.

• msearch

  Standalone one-shot multi-source literature search. Takes a single query and returns one merged, ranked result set + synthesized landscape from the internet (web + arXiv), local channels (Zotero, Obsidian, local PDFs), and the cloud mechanic-db SEARCH service. Same retrieval engine as `/research-lit

• next-round

  Between-round transition helper for multi-round /auto exploration. Archives the just-finished round's outputs into rounds/round_<N>/ and drafts the next round's task.md with recommended behavior-source/mechanism params, by reading the orchestrator-owned Global Exploration Memory (research_memory.jso

• notify

  Draft a research-progress briefing and dispatch it through whatever notification service the user has already configured. Channel-agnostic — this skill only drafts the briefing text and scans for a configured service; it does not hard-code or recommend any particular channel or tool. Opt-in: it does

• novelty-check
• paper-figure
• research-lit
• research-refine-pipeline
• research-refine
• research-review
• result-to-claim
• run-experiment
• training-check

Subagents (4)

• claim

  The claim agent of /auto. Runs the `/auto-claim` skill under two orthogonal axes — `BEHAVIOR_SOURCE` (given / given-validation / discovery) sets where the behavior comes from and whether it is validated; `MECHANISM` (given / discovery) sets who picks the mechanism method. `discovery` generates ranke

• experiment

  The experiment agent of /auto. Wraps the /auto-experiment skill, which folds mechanism-family routing inline before implementing, code-reviewing, and deploying the experiment suite. Supports two-step invocation — first call returns candidate families for the orchestrator's mini-prompt, second call (

• iteration

  The iteration agent of /auto. Runs the /auto-iteration-loop skill — an autonomous review loop that consumes /auto-verify's four-state output (PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS) plus the orthogonal deferred bucket and routes each claim to the right back-edge (① variant-only fix / ②

• verify

  The verify agent of /auto. Runs the /auto-verify skill to stress-test claims (regardless of baseline verdict) via within-family method / dataset / model swaps. Two mandatory integrity gates — Phase 2 per-claim baseline audit (runs for every target claim) and Phase 9 per-claim variant audit on Phase

MCP servers (2)

• llm-chat

  runs: uv run --with-requirements ${CLAUDE_PLUGIN_ROOT}/mcp-servers/llm-chat/requirements.txt ${CLAUDE_PLUGIN_ROOT}/mcp-servers/llm-chat/server.py

• mechanic-db

  runs: uv run --with-requirements ${CLAUDE_PLUGIN_ROOT}/mcp-servers/mechanic-db/requirements.txt ${CLAUDE_PLUGIN_ROOT}/mcp-servers/mechanic-db/server.py

Similar plugins