6 ms·
Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. (Though these days a lot of businesses offer unoffici
by andai 3y ago
Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. (Though these days a lot of businesses offer unofficial 3rd party APIs by scraping.)
It's also stupid and massively wastes computation and traffic on both ends.
Then people would think "but then I'd have to pay money to offer a free API", but they're already doing that in a more expensive way via the web interface anyway.
- gabipurcaru 3y ago> people only do it when there's no API Legitimate scrapers, maybe. Everyone else does it to circumvent the API limitations, by posing as real traffic. APIs imply API keys which can be traced and banned.
- yjftsjthsd-h 3y agoAnd what makes a scraper legitimate?
- ben0x539 3y agoWell, that's what happens when you make the API more limited than what people can do with scraping (or with using the public "our own web client" API key).
- thejohnconway 3y agoNo, there’s a huge number of bots out there just doing stupid hacky stuff like trying for wp/admin/. Vast numbers of what I’d characterise as script-kiddy type attempts. Looking at the raw logs of a webserver not behind something like CloudFlare really opened my eyes to this. There’s an extraordinary number of garbage requests.
- pixl97 3y agoYea, there was an HN thread complaining about WAF's a few days ago, but damn, if you're on the public internet having some kind of WAF cuts down the bullshit responses by a massive amount. Same thing goes for email. I had one particular domain I managed spam filtering on and for every 10,000 messages we received, 1 was delivered to users. The amount of crap requests on the internet is insane.
- kube-system 3y agoAnd that's just requests on 80/443. Hook up an IDS to the WAN interface and, you'll see thousands of attempts per day to exploit vulnerabilities across other services too. My low-traffic personal projects get 90%+ malicious traffic.
- gumballindie 3y agoI think a more efficient approach would be to prosecute those who do it - bandwidth costs money - and to implement mechanisms to deter them. A not insignificant number of ai workers think are entitled to using people’s bandwidth, servers and intellectual property for free. Until a couple of such people are taken to task this wont stop.
- JohnFen 3y ago> Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. Many sites intentionally make scraping as painful as possible because they don't want anyone scraping them.