starlake-skills

Plugin starlake-skills in starlake-ai/starlake-skills. Commands read from the repo's manifest — 1 month ago.

Skills (73)

• acl-dependencies

  Visualize access control with `starlake acl-dependencies` — graph which users and groups can access which tables, from the ACL and RLS (row-level security) definitions in your YAML, output as SVG, PNG, or JSON

• autoload

  Infer schemas from files in the incoming directory and load them into the warehouse in one step with `starlake autoload`. Use for quick starts or new data files with no YAML table definitions yet; generates the table configs automatically.

• bootstrap

  Create a new Starlake project with `starlake bootstrap` — scaffold the standard metadata/load/transform directory structure and starter config from a template (quickstart, simple). Use when initializing or setting up a new project.

• bq-info

  Get BigQuery table information with `starlake bq-info` — schema, row count, size, partitioning, and clustering details for datasets and tables in a GCP project

• cnxload

  Load a Parquet, CSV, or JSON file directly into a JDBC database table with `starlake cnxload`. Use to push one file into PostgreSQL, MySQL, or any JDBC target without the full Starlake load pipeline or YAML schemas.

• col-lineage

  Trace column-level lineage for one transform task with `starlake col-lineage` — show how each output column derives from its source columns through the SQL, for fine-grained data provenance and impact analysis

• compare

  Diff two versions of a Starlake project with `starlake compare` — report schema changes, added/removed tables, and modified configurations between file paths, git commits, or git tags

• config

  Starlake configuration reference — `SL_*` environment variables, `application.sl.yml` structure, project layout, attribute types, and storage patterns. Use for cross-cutting configuration questions and as the index to the specialized skills (connection, load, secure, expectations, dag-generate...).

• connection

  Create or modify database connection definitions in `application.sl.yml` (BigQuery, Snowflake, DuckDB, DuckLake, PostgreSQL, Redshift, Kafka, Elasticsearch). Use to add or configure a connection; verify it with `starlake settings --test-connection`.

• console

  Start the interactive REPL with `starlake console` — a shell/prompt session that runs Starlake commands without restarting the JVM each time, for fast iterative work on loads and transforms

• dag-create

  Create a ready-to-deploy one-off orchestration DAG (Airflow or Dagster Python file) from a natural-language pipeline description via the starlake-orchestration API — a bespoke single file, no Jinja2 template and no `starlake dag-generate` run needed

• dag-deploy

  Deploy generated DAG files with `starlake dag-deploy` — copy DAGs produced by dag-generate into a target directory such as your Airflow DAGs folder or Dagster repository, optionally cleaning it first. Use to publish or ship DAGs to the scheduler

• dag-generate

  Generate orchestration DAGs with `starlake dag-generate` — render Airflow or Dagster Python DAG files from your project metadata and Jinja2 templates to schedule loads and transforms; regenerate after schema or schedule changes

• dag-template-generate

  Generate a custom Jinja2 DAG template and its YAML DAG configuration for `starlake dag-generate` from a natural-language pipeline description (Airflow, Dagster, Snowflake) — use when built-in templates need extra tasks or custom behavior

• esload

  Load or index data into Elasticsearch with `starlake esload` (alias `starlake index`) — push Parquet or JSON datasets into Elasticsearch indices with custom mappings, document IDs, and timestamp-based index names.

• expectations

  Define data quality checks and assertions on loaded or transformed data — SQL-based expectations (uniqueness, row counts, null checks) evaluated by `starlake load` and `starlake transform`, with warn-or-fail thresholds

• extract-bq-schema

  Extract table schemas from BigQuery datasets with `starlake extract-bq-schema`: reverse-engineer BigQuery tables into Starlake YAML via the native BigQuery API (not JDBC), for a whole GCP project, specific dataset.table pairs, or external datasets

• extract-data

  Extract data only (no schema) from JDBC database tables into CSV/Parquet files with `starlake extract-data`: full or incremental export, parallel partitioned extraction, and schema/table include-exclude filtering

• extract-rest-data

  Extract data only from REST API endpoints into CSV or JSON Lines files with `starlake extract-rest-data`: pagination, auth (bearer, API key, OAuth2), rate limiting, incremental extraction, resume, proxy, and mTLS support

• extract-rest-schema

  Infer table schemas from REST API endpoints with `starlake extract-rest-schema`: fetch sample JSON responses and reverse-engineer them into Starlake YAML domain/table definitions (schema only; use extract-rest-data for the rows)

• extract-schema

  Extract schema only from a JDBC database with `starlake extract-schema`: reverse-engineer tables, columns, and types from Postgres, MySQL, Oracle, DuckDB, etc. into Starlake YAML load configs (no data export)

• extract-script

  Generate custom extraction and DDL scripts (CREATE, ALTER, DROP) from Mustache/SSP templates with `starlake extract-script`, per database platform (BigQuery, Postgres, Synapse), with audit schema and incremental delta-column support

• extract

  Extract both schema AND data from a JDBC database in one shot with `starlake extract`: reverse-engineer tables into Starlake YAML then export the rows to files; also drives REST API and OpenAPI/Swagger extract configs

• freshness

  Check data freshness with `starlake freshness` — detect stale data by querying last updated timestamps per table, with warn/error thresholds, persisted to the SL_LAST_EXPORT audit table for monitoring

• iam-policies

  Apply IAM (Identity and Access Management) policies with `starlake iam-policies` — set up permissions, roles, ACLs, and policy tags on warehouse resources such as BigQuery datasets and tables

• index

  Index data in Elasticsearch with `starlake index`, an alias for `starlake esload` — same options for loading Parquet or JSON datasets into Elasticsearch indices with mappings, document IDs, and timestamp-based index names.

• infer-schema

  Infer a Starlake table schema from a data file with `starlake infer-schema`: detect column names and types from a CSV, JSON, XML, or Parquet sample (or a JSON Schema) and generate the load YAML config

• ingest

  Ingest specific file paths into one domain/table with `starlake ingest <domain> <schema> <paths>` — low-level, programmatic loading of named files. Use when targeting exact files; use load to scan the pending area by directory.

• job

  Run a SQL or Python transformation task with `starlake job`, an alias for `starlake transform` — identical options and behavior for executing, compiling, or previewing transform tasks and their dependencies

• kafkaload

  Load files into Kafka topics or offload Kafka topics to files with `starlake kafkaload`, in batch or streaming mode with optional SQL transforms. Use to publish data to Kafka, dump a topic to Parquet, or stream between them.

• lineage

  Visualize data lineage with `starlake lineage` — a task dependency graph showing data flow and upstream/downstream dependencies between transform tasks and source tables, output as SVG, PNG, DOT, or JSON

• load

  Load files from the pending area into the data warehouse with `starlake load`, validating against defined table schemas and applying write strategies. Use when YAML schemas already exist; see autoload to infer schemas from incoming files.

• metrics

  Compute statistics and profile table data with `starlake metrics` — min/max/mean/stddev for continuous columns, distinct counts and category frequencies for discrete ones, stored in the SL_METRICS audit table

• migrate

  Migrate a Starlake project to the latest version format with `starlake migrate`: upgrade YAML config files after a Starlake upgrade, renaming deprecated keys and handling breaking schema changes between releases

• parquet2csv

  Convert Parquet files to CSV with `starlake parquet2csv` — export data to systems without Parquet support or inspect it in human-readable form, with configurable separators, headers, and partitions

• preload

  Check whether files are ready for loading with `starlake preload` — counts stageable files in the incoming/landing area (imported), staged files in the pending area (pending), or tests an ack file (ack). Use in orchestration to gate stage/load steps.

• quack

  Manage Quack DuckDB query servers with `starlake quack` — expose DuckLake tables to remote clients over a thin protocol without sharing storage credentials; serve (foreground), start/stop/list/stop-all (background)

• secure

  Apply row-level security (RLS) and column-level security (CLS) with `starlake secure` — mask PII, hash or encrypt sensitive columns, and restrict which users or groups see which rows, columns, and tables

• semantic-export

  Export semantic models from metadata/semantic to Apache Ossie, Looker LookML, or Power BI TMDL with the starlake semantic-export command

• semantic

  Author semantic model YAML in metadata/semantic for BI tools and AI agents — logical tables, dimensions, facts, metrics, relationships, and verified queries, exportable via `starlake semantic-export`

• serve

  Run the Starlake HTTP server with `starlake serve` — expose all CLI commands as a REST API for programmatic or remote execution, with configurable host and port. Use for server mode, API access, or daemon-style usage

• settings

  Print resolved project settings or test an existing connection with `starlake settings --test-connection <name>`. Use to debug configuration or check that a BigQuery, Snowflake, DuckDB, or PostgreSQL connection works; use connection to define one.

• site

  Generate a documentation website for the project with `starlake site` — docs covering all domains, tables, attributes, transforms, and their relationships, output as Docusaurus MDX or JSON

• stage

  Move data files from the landing area to the pending area with `starlake stage`, decompressing archives and handling ACK files. Use as the first ingestion step before `starlake load` processes the staged files.

• summarize

  Display table summary statistics with `starlake summarize` — quick row counts, column statistics, and sample data for exploring or profiling a table in a domain

• table-dependencies

  Generate a table dependency graph with `starlake table-dependencies` — an entity-relationship (ER) diagram of your data model built from the foreign key relationships declared in table YAML, output as SVG, PNG, or JSON

• transform

  Run a SQL or Python transformation task with `starlake transform` — execute, compile, or preview transform queries, run a task with its upstream dependencies, dry-run on BigQuery, or run all tasks matching a tag

• validate

  Check project YAML and config validity with `starlake validate` — syntax, JSON Schema compliance, type/connectionRef/dagRef/foreign-key references, write-strategy consistency. Use to catch config errors before runtime, not to test live connections.

• xls2yml

  Convert Excel to YAML with `starlake xls2yml`: turn domain/table schema definitions managed in .xlsx spreadsheets into Starlake load configs, with optional ACL/RLS policy files and BigQuery IAM policy tags

• xls2ymljob

  Convert Excel job definitions to YAML with `starlake xls2ymljob`: turn transform job/task specs managed in .xlsx spreadsheets into Starlake task configs (use xls2yml for domain/table schemas)

• yml2ddl

  Generate SQL DDL (CREATE/ALTER TABLE) from Starlake YAML table definitions with `starlake yml2ddl`: target BigQuery, Snowflake, Postgres, Redshift, Synapse, or DuckDB via type mappings, optionally applying directly to the database

• yml2xls

  Convert YAML to Excel with `starlake yml2xls`: export Starlake domain/table definitions to .xlsx spreadsheets for sharing data models with non-technical stakeholders or round-tripping back via xls2yml

• starflow-builder

  Scaffold custom Starflow personas and workflow skills, or customize existing ones via the layered config. Use when the user says "create a persona", "new starflow skill", "extend starflow", or "starflow builder".

• starflow-code-review

  Adversarial multi-persona review of data pipeline changes. Use when the user says "review pipeline", "code review", or "review this data change", or wants pipeline configs, transforms, and expectations checked before merging.

• starflow-create-data-architecture

  Design the overall data architecture including layers, storage, engines, and governance. Use when the user says "create data architecture" or "design the data platform".

• starflow-create-pipeline-spec

  Create a complete pipeline specification covering extract, load, transform, and orchestrate. Step-file workflow with resumable progress. Use when the user says "create pipeline spec" or "design a data pipeline".

• starflow-data-analyst

  Business Data Analyst agent (Lea): guides domain discovery and source analysis. Use when the user says "data-analyst" or "talk to the data-analyst", or asks to map data domains and sources, analyze a source system, or find out what data exists and who owns it.

• starflow-data-architect

  Data Architect agent (Winston): designs data platforms, schemas, and pipeline architecture. Use when the user says "data-architect" or "talk to the data-architect", or asks for help designing warehouse layers, table schemas, a pipeline spec, or a semantic model.

• starflow-data-engineer

  Data Engineer agent (Amelia): builds and maintains ETL/ELT pipelines with Starlake. Use when the user says "data-engineer" or "talk to the data-engineer", or asks to implement a pipeline, turn a pipeline spec into Starlake YAML, or debug a load/transform run.

• starflow-data-quality-engineer

  Data Quality Engineer agent (Quinn): ensures data integrity with expectations, lineage, and governance. Use when the user says "data-quality-engineer" or "talk to the data-quality-engineer", or asks to add data quality checks, review expectations, or trace lineage.

• starflow-data-quality-review

  Review and design data quality expectations for Starlake pipelines. Use when the user says "review data quality" or "check expectations".

• starflow-dev-pipeline

  Implement a data pipeline from a pipeline specification, generating Starlake configuration files. Use when the user says "implement pipeline" or "dev this pipeline".

• starflow-domain-discovery

  Discover and document data domains, sources, and ownership. Use when the user says "discover data domains" or "map data sources".

• starflow-help

  Starflow orientation: tells the user where they are in the methodology and recommends the next step. Use when the user asks "what should I do next", "starflow help", "where am I in starflow", or just "/help" inside Starflow.

• starflow-lineage-review

  Review and document data lineage across pipeline stages. Use when the user says "review lineage" or "trace data flow".

• starflow-orchestration-design

  Design orchestration DAGs for scheduling and managing data pipeline execution. Use when the user says "design orchestration" or "create DAG configuration".

• starflow-platform-engineer

  Platform Engineer agent (Max): manages infrastructure, orchestration, and deployment for data pipelines. Use when the user says "platform-engineer" or "talk to the platform-engineer", or asks about deploying DAGs, orchestration setup, connections, or environments.

• starflow-retrospective

  End-of-epic retrospective for a data pipeline batch. Use when the user says "run a retro", "epic retrospective", or "retro the {epic}", or wants to close out an epic with lessons learned and action items.

• starflow-schema-design

  Design Starlake-compatible table schemas with types, constraints, privacy, and expectations. Use when the user says "design schema" or "create table definition".

• starflow-semantic-model-design

  Design a business semantic model (dimensions, facts, metrics, relationships, verified queries) over pipeline tables and write it to metadata/semantic. Use when the user says "design semantic model", "semantic layer", or "expose tables to BI/AI".

• starflow-source-analysis

  Analyze data sources in depth: schema, quality, volume, and extraction strategy. Use when the user says "analyze data source" or "profile this data source".

• starflow-sprint-planning

  Plan and track sprint progress for data pipeline implementation. Use when the user says "sprint planning" or "plan data sprint".

• starflow-transform-design

  Design SQL transformations for data pipelines with quality checks and dependency management. Use when the user says "design transforms" or "create SQL transformations".

Similar plugins