It is harmful because innocent users routinely get caught in your dragnet. And why even have a public website if the goal is not to serve it? What is the actua

It is harmful because innocent users routinely get caught in your dragnet. And why even have a public website if the goal is not to serve it? What is the actual problem with serving users? You mentioned incredible load. I would stop using inefficient PHP or JavaScript or Ruby for web servers. I would use Go or Rust or a comparable efficient server with native concurrency. Survival always requires adaptation. How do you know that the alleged proxies belong to the same scrapers? I would look carefully at the values contained in the IP chain as determined by XFF to know which subnets to rate-limit as per their membership in the XFF. Another way is to require authentication for expensive endpoints.