shub-workflow-toolkit

Plugin shub-workflow-toolkit from marketplace shub-workflow in scrapinghub/shub-workflow. Commands read from the repo's manifest — 2 days ago.

Skills (10)

• scanjobs-programs

  Use for help using shub_workflow's scanjobs tool — scanning ScrapyCloud jobs to extract and plot data from stats, logs, items, spider args or job metadata. Also reach for it, instead of a hand-rolled job-listing script, to take a census of a spider's jobs over a time window: how many ended with each

• shub-workflow-architectures

  Use when a developer wants help designing (or redesigning) the architecture of a shub-workflow based Scrapy Cloud project — "help me design the architecture for this project", "how should I structure the crawl managers / consumers / delivery", "which components do I need", "how do I feed my spiders

• shub-workflow-crawl-managers

  Use when building, updating, fixing, or understanding a shub-workflow crawl manager — a script that schedules spider jobs on Scrapy Cloud and reacts to their outcomes, built on shub_workflow.crawl (CrawlManager / PeriodicCrawlManager / GeneratorCrawlManager / AsyncSchedulerCrawlManagerMixin) and Wor

• shub-workflow-diagrams

  Use when asked to build, update or restyle an architecture diagram for a shub-workflow based project — a workflow/pipeline diagram (the spiders, crawl managers, consumers, issuers, deduplicators, balancers, downloaders, delivery and the storages between them) or a class hierarchy / UML diagram of it

• shub-workflow-fshelper

  Use when reading, writing, listing, copying, moving, or deleting files through shub-workflow's cloud-agnostic filesystem layer — the `shub_workflow.utils.futils` module and its `FSHelper` class (the recommended one; `S3Helper` is its deprecated backward-compatibility base) — i.e. any code that touch

• shub-workflow-graph-managers

  Use when building, updating, fixing, or understanding a shub-workflow graph manager — a script that runs an arbitrary DAG of tasks (spiders and scripts, including crawl managers and deliver scripts) on Scrapy Cloud with dependencies, built on shub_workflow.graph (GraphManager + Task / SpiderTask) an

• shub-workflow-hcf

  Use when working with a ScrapyCloud Hubstorage Crawl Frontier (HCF) from a shub-workflow project — running or extending the standalone frontier tool `shub_workflow.utils.hcf.FrontierTool` (list slots, count, dump, add seeds, delete batches or slots, move a slot into another), installing the `with-hc

• shub-workflow-issuers

  Use when building, updating, fixing, or understanding a shub-workflow issuer — a fundamental Scrapy Cloud data-pipeline component that reads a massive input (finished spider jobs or batch files), processes/dedups/transforms the items, and writes a massive batch output, chaining into the post-crawl p

• shub-workflow-monitors

  Use when building, updating, fixing, or understanding a shub-workflow monitor — a script that gives a broad cross-job view of a workflow: scanning the spider/script jobs in a time window, aggregating their stats (and log-derived stats), computing ratios, emitting reports, and raising threshold alert

• shub-workflow-scripts

  Use when writing, fixing, or updating a Python script built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin / ArgumentParserScript) — i.e. any script that operates on Scrapy Cloud: scheduling spiders or scripts, scanning/querying SC jobs, aggregating

Similar plugins