4 ms·
>so if you don't want to get sued or blacklisted, you can do no kind of (detectable) scraping. So why would it be hard to make scraping undetectable, anyway, u
by gorkonsine 9y ago
>so if you don't want to get sued or blacklisted, you can do no kind of (detectable) scraping.
So why would it be hard to make scraping undetectable, anyway, unless you do it particularly incompetently? In theory, it seems pretty easy: use a browser string that matches an existing popular browser, and make sure to not load anything faster than a human would.
- tomarr 9y agoHave you done much scraping in the past? There's normally a lot more to it when javascript is involved, captcha systems etc. This is obviously helped recently by the relatively new headless modes for Chrome & Firefox, but before that it was using buggy headless implementations or Selenium. These weren't well suited to operations at scale.