3 ms·
That's a feature if you ask me. The whole point of scraping websites is to get the data I want while discarding noise like interface chrome and advertising dire
by matheusmoreira 6y ago
That's a feature if you ask me. The whole point of scraping websites is to get the data I want while discarding noise like interface chrome and advertising directly to the garbage.
If they'd like me to pay for access, they should return HTTP 402 Payment Required instead of letting me download the page for free. Perhaps they could also rate limit the network connection to prevent denial of service. Why straight up block automated user agents though? That sucks.
- octoberfranklin 6y ago> they should return HTTP 402 Payment Required That's illegal. They must first return HTTP 418 Know Your Customer Required.