3 ms·
On the one hand, I do think that scraping should be legal, on the other hand, I maintain a lot of websites and scrapers are a gigantic pain in the ass that make
by martin82 4y ago
On the one hand, I do think that scraping should be legal, on the other hand, I maintain a lot of websites and scrapers are a gigantic pain in the ass that make hosting a popular site for normal people cost-prohibitive (you need to scale up servers or invest engineering time to prevent scraping).
This can only lead to a more centralised web where only rich people can afford to host content.
There needs to be a solution like some kind of internet metadata archive, distributed on IPFS or something where bots can get their data if they want.
- account42 4y agoWith what kind of sites do you see this as as a problem and what level of traffic do you get from these scrapers? There was a discussion [0] 11 days ago about this concerning a search engine, but for many other websites you can just pre-generate almost all of the content and use microcaching to cover anything that has to be dynamic to drive the per-request cost to almost 0. [0] https://news.ycombinator.com/item?id=32339314 https://news.ycombinator.com/item?id=32339314