← All projects

scrapescale-chef

Chef cookbook + knife-solo runner that provisions the scrapescale scrapyd EC2 host.

Infra-as-code companion to the scrapescale spider fleet - so the production scraper host could be rebuilt from a clean EC2 image in one command, rather than being a snowflake server nobody dared touch. Uses Chef with knife-solo (Ruby) to bootstrap a fresh node into a working scrapyd worker: SSH in, copy cookbooks, run chef, configure the proxy in the shell.

Single-host setup - one cookbook, one target node defined in YAML - so the scrapescale side of the CRE stack was reproducibly provisioned rather than a hand-built snowflake.