3 ms·
Why can't they just allow automated user agents? I should be able to scrape websites if I want to. Why do user agents have to be browsers?
by matheusmoreira 6y ago
Why can't they just allow automated user agents? I should be able to scrape websites if I want to. Why do user agents have to be browsers?
- entangledqubit 6y agoThis doesn't mix well with the ad-based compensation model. Sadly, there still doesn't seem to be much in the way of micropayment infrastructure.
- octoberfranklin 6y ago> there still doesn't seem to be much in the way of micropayment infrastructure. Anti Money Laundering regulation killed it: KYC doesn't scale down to micropayment levels. If you want to fix the web, you have to roll back the AML/KYC insanity. Until that happens, the web will stay broken, because paying with attention (ads) is magically exempt from the AML/KYC insanity, whereas paying with money or anything money-equivalent (fungible and transferable) is not.
- matheusmoreira 6y ago> the AML/KYC insanity Just finished reading about this and I completely agree. I can't imagine having a company and being literally obligated by law to violate everything I personally believe in about privacy and freedom just to help the government be even more efficient at marginalizing people.
- matheusmoreira 6y agoThat's a feature if you ask me. The whole point of scraping websites is to get the data I want while discarding noise like interface chrome and advertising directly to the garbage. If they'd like me to pay for access, they should return HTTP 402 Payment Required instead of letting me download the page for free. Perhaps they could also rate limit the network connection to prevent denial of service. Why straight up block automated user agents though? That sucks.
- octoberfranklin 6y ago> they should return HTTP 402 Payment Required That's illegal. They must first return HTTP 418 Know Your Customer Required.
- megous 6y agoExactly, or be able to just use a text-mode browser. Or wget to save a set of pages for later. I understand protecting commenting with captcha, or contact forms. But captcha on regular read-only access to public web pages in the style of Cloudflare is a bit ridiculous. One thing contact forms should have is a static indication there's a captcha in use. I've filled all too many forms that just sent my written text to void, because I block some domains.
- lovegoblin 6y agoBeing scraped isn't free, if it's at a large enough scale. Plus, it's not just benign read-only scrapers. Have you looked at the spam folder of your email recently? That's what every comment section and user bio and god knows what else would look like if you just blindly allow all automated traffic.