28 ms·
Why is it useless and harmful? Many of us are struggling—without massive budgets or engineering teams—to keep services up due to incredible load from scrapers i
by aorth 1y ago
Why is it useless and harmful? Many of us are struggling—without massive budgets or engineering teams—to keep services up due to incredible load from scrapers in recent years. We do use rate limiting, but scrapers circumvent it with residential proxies and brute force. I often see concurrent requests from hundreds or thousands of IPs in one data center. Who do these people think they are?
- OutOfHere 1y agoIt is harmful because innocent users routinely get caught in your dragnet. And why even have a public website if the goal is not to serve it? What is the actual problem with serving users? You mentioned incredible load. I would stop using inefficient PHP or JavaScript or Ruby for web servers. I would use Go or Rust or a comparable efficient server with native concurrency. Survival always requires adaptation. How do you know that the alleged proxies belong to the same scrapers? I would look carefully at the values contained in the IP chain as determined by XFF to know which subnets to rate-limit as per their membership in the XFF. Another way is to require authentication for expensive endpoints.
- DamonHD 1y ago[flagged]
- OutOfHere 1y agoThe truth is often unpleasant, and it owes you nothing. I ask you in return to be more open to it. By wanting to hide and suppress the truth, it is you who is not nice.
- DamonHD 1y agoYou are wrong and coming across entirely pompous. I see what I see on my servers, and you clearly have not seen the same.
- immibis 1y agoResidential proxy users are paying on the order of $5 per gigabyte, so send them really big files once detected. Or "click here to load the page properly" followed by a trickle of garbage data.