3 ms·
Forgetting about law and copyright and robots.txt because hey most scrapers have to rely on fair use anyway, you even forget about basic consideration for the s
by natch 2y ago
Forgetting about law and copyright and robots.txt because hey most scrapers have to rely on fair use anyway, you even forget about basic consideration for the sites you hammer.
That sounds like a scold, but it's meant as an observation.
Now I will embed some implied scolding in what's to follow, but feel free to ignore that part; I wouldn't expect you to care.
But if you lack even a shred of human decency or morals, perhaps there's one more reason you might consider for spreading out your requests across sites and time, instead of absolutely pushing the abuse to the hilt, and that is that if you change what you are doing, and take a slower, more gentle approach, and I'm appealing to your selfishness here because clearly that is the only viable way into your head, then you will be less likely to cause countermeasures, and more likely to succeed.
- marcell 2y agoIt’s actually a Chrome extension, and it runs in the user’s browser. You can configure the scraping rate and the app warns against higher rates (more than 5 tabs at a time). I don’t think any sites are going to get hammered though, even at the fastest rates. The limiting factor is often LLM token rates.
- natch 2y agoThanks for the clarification.