3 ms·
There's a project called woob (https://woob.tech/ https://woob.tech/) that implements python scripts that are a little bit like scrapers but only 'scrape' on de
by cogburnd02 5y ago
There's a project called woob (https://woob.tech/ https://woob.tech/) that implements python scripts that are a little bit like scrapers but only 'scrape' on demand from requests from console & desktop programs.
How much of this article do you think would apply to something like that? e.g. something like 'wait a second (or even two!) between successive hits' might not be necessary (one could perhaps shorten it to 1/4 second) if one is only doing a few requests followed by long periods of no requests.
- cushychicken 5y agoInteresting question. My first instinct is to say that woob seems closer in use case to a browser than a scraper, as it seems largely geared towards making rich websites more easily accessible. (If I'm reading the page right, anyway.) A scraper is basically just hitting a web page over and over again, as fast as you can manage. The trick, IMO, is to be closer to browser level loading on a server than scraper. Make sense?