6 ms·
Maybe it’s a better idea to start or subscribe to an ethical scraping initiative with some user agent header that references some kind of policy and verifiable
by pacifika 5y ago
Maybe it’s a better idea to start or subscribe to an ethical scraping initiative with some user agent header that references some kind of policy and verifiable membership id
- intev 5y agoI think it's a pretty safe assumption that if everyone started scraping with certain headers almost every website will blanket block those headers.
- weird-eye-issue 5y agoNaive
- meltedcapacitor 5y agoThat's the existing default: if a service doesn't mind scraping, they need not take any countermeasure (other than simple rate limiting with a clear status code). Any site that does go beyond that is telling data thieves that they're not welcome.