21 ms·
Oh, I see what you mean. Yeah, there's definitely a lot to be said for breaking up your scraped data into separate files rather than having it all in a single
by simonw 3y ago
Oh, I see what you mean.
Yeah, there's definitely a lot to be said for breaking up your scraped data into separate files rather than having it all in a single file.
I have a few projects where I do that kind of thing. My best example is probably this one, where I scrape the "--help" output for the AWS CLI commands and write that into a separate file for each command:
help-scraper/tree/main/aws: https://github.com/simonw/help-scraper/tree/main/aws https://github.com/simonw/help-scraper/tree/main/aws
This is fantastically useful for keeping track of which AWS features were added or changed at what point.