4 ms·
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape. You could perhaps even get website opera
by jay_kyburz 3mo ago
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
- jay_kyburz 3mo agoHow about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
- mips_avatar 3mo agothat's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
- jay_kyburz 3mo agoI was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
- mips_avatar 3mo agoYeah. Though I guess the point I thought of was like a deals site. That would have infinite pages and content
- Symbiote 3mo agoI have essentially this at work, but the scrapers ignore it. (Or at least many, many scrapers ignore it.)
- jay_kyburz 3mo agoIt wont work unless everybody does it because otherwise it's more work for the scrapers not less. They need to implement two systems instead of one. And we'll never get everybody to do it.
- flaburgan 3mo agoWell this is not what is happening in practice, Wikipedia / Wikidata, OpenStreetMap, OpenFoodFacts... All provide APIs and even a full dump of their database available to download for free, but no, the stupid bots still DDoS them 24h/24.
- pocksuppet 3mo agoWhy don't they take legal action?
- jay_kyburz 3mo agoThere is nobody to sue. The traffic is coming from millions of residential IPs.
- inigyou 3mo agoWhy would that stop them?