3 ms·
Bot abuse is a huge problem for search engines and similar services. Modern scrapers are virtually impossible to detect and abusing publicly available endpoints
by marginalia_nu 4y ago
Bot abuse is a huge problem for search engines and similar services. Modern scrapers are virtually impossible to detect and abusing publicly available endpoints for free compute is basically an industry today.
It's a real shame. Means everything goes behind a signup wall.
For reference, about 1% of the traffic I get on my site is probably human (20k quries/day). An attemted 2 million queries/day from confirmed bots.
- rchaud 4y agoBot problems are dealt with via captchas. The sign-in requirements are coming up because the era of cookie-based analytics is ending; iOS already blocks many by default, and a third of internet users have adblocking on anyway. Logged-in user activity is significantly easier to to track than cookie-based tracking was.
- siva7 4y agoI would say that's wishful thinking. Using only captchas doesn't work anymore if you are specially targeted and also a reason why we are seeing nowadays a combination of sign-in/phone verification/captchas to combat the bot problematic.