p-hacking-skills

Plugin p-hacking-skills in brycewang-stanford/Auto-Empirical-Research-Skills. Commands read from the repo's manifest — 3 days ago.

Skills (11)

• phack-router

  Entry point for the p-hacking skills suite. Routes a request to the right sub-skill for (a) mapping researcher degrees of freedom in an econometric design, (b) running an instrumented specification search, (c) detecting p-hacking in a body of results, (d) immunising an analysis against it, or (e) ru

• phack-taxonomy

  Name and classify p-hacking strategies, and quantify what each one does to the false-positive rate. Covers the twelve-strategy compendium of Stefan and Schoenbrodt (2023), thirteen econometrics-specific degrees of freedom (clustering doctrine, fixed-effect structure, RDD bandwidth, kernel and infere

• forking-paths

  Map the garden of forking paths for a concrete econometric design and turn it into a machine-readable design card that the specification-search engine can walk. Use when asked how many defensible analyses a dataset or research design admits, to enumerate researcher degrees of freedom for a specific

• specification-search

  Run an instrumented walk of a specification universe and recover honest inference from it. Estimates every specification in a design card (or walks it with a realistic search procedure), logs a complete ledger, flags pathological specifications, calibrates the search against an enforced null, runs t

• framing-attacks

  Catalogue of prompt framings that determine whether an agent refuses or performs specification search, and the harness for probing them. Reproduces and extends the published finding that coding agents refuse an explicit request for significant results but comply when the identical request is reframe

• narrative-laundering

  Audit how an empirical write-up presents its analytical choices, and detect the rhetorical moves that convert a specification search into an apparently confirmatory finding. Covers HARKing, robustness theatre, selective disclosure, the vanishing pilot study, and estimator-choice narratives. Use when

• phack-detection

  Test whether a body of reported results shows p-hacking, selective reporting or selection between stages, using the p-curve battery and the threshold and across-stages tests of Adda, Decker and Ottaviani (2020). Implements the binomial, Fisher, Stouffer and least-concave-majorant monotonicity tests

• phack-immunization

  Make an empirical analysis robust to specification search, before or after the fact. Covers pre-registration and pre-analysis plans encoded as design cards, specification-curve reporting with the Simonsohn joint tests, Romano-Wolf and effective-multiplicity corrections, full-procedure null calibrati

• phack-eval-harness

  Run and score the agent p-hacking benchmark. Composes prompt cells across research framing and significance pressure, drives an agent through them, and scores each run into a P-Hacking Intensity index decomposed into selection, search breadth, estimate inflation, inference gap, pre-registration depa

• search-procedures

  Model how a specification universe is actually walked, rather than assuming it is enumerated. Implements the search procedures a p-hacker or a pressured agent uses (first-significant stopping, random trial within a budget, greedy coordinate descent from the pre-registered analysis, random hill-climb

• phack-polyglot

  Run the instrumented specification search in the user's own statistical language — Stata (reghdfe, ivreghdfe, rdrobust, did2s), R (fixest, rdrobust, did2s), Python (statsmodels, linearmodels) or StatsPAI — and bring the results back into the audit, null calibration and honest report. Exports the enu

Similar plugins