4 ms·

There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffi

There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.

sporp 2mo ago
How did you implement this? My site's getting hammered, any tips would be appreciated
- ComputerPerson 2mo ago
  What kind of uptime/expectations/etc. are you working with? Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss. Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users. We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
- subnetcap 2mo ago
  [flagged]
- configdesk 2mo ago
  [flagged]
- broker_desk 2mo ago
  [flagged]
cogman10 2mo ago
One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1] [1] https://www.gentoo.org/news/2026/01/05/new-year.html https://www.gentoo.org/news/2026/01/05/new-year.html
- ComputerPerson 2mo ago
  Basically volunteer-run. Which makes sense in this context; what gets worked on is mostly about what people want to work on.
Symbiote 2mo ago
> Most scrapers are relatively honest in some way shape or form. Did you miss a "dis" in there?
gruez 2mo ago
>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?
- ComputerPerson 2mo ago
  One, their functionality is bad, two, we don't want to block robot traffic. We're redicting it to a separate sever so that overly-large waves don't bring down our main servers. We definitely want to be scraped. To your point: I think you're simply mistaken that they offer a good scraper-detection solution, but I'm all ears.

sporp 2mo ago

How did you implement this? My site's getting hammered, any tips would be appreciated

ComputerPerson 2mo ago
What kind of uptime/expectations/etc. are you working with? Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss. Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users. We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
subnetcap 2mo ago
[flagged]
configdesk 2mo ago
[flagged]
broker_desk 2mo ago
[flagged]

cogman10 2mo ago

One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1] [1] https://www.gentoo.org/news/2026/01/05/new-year.html https://www.gentoo.org/news/2026/01/05/new-year.html

ComputerPerson 2mo ago
Basically volunteer-run. Which makes sense in this context; what gets worked on is mostly about what people want to work on.

Symbiote 2mo ago

> Most scrapers are relatively honest in some way shape or form. Did you miss a "dis" in there?

gruez 2mo ago

>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?

ComputerPerson 2mo ago
One, their functionality is bad, two, we don't want to block robot traffic. We're redicting it to a separate sever so that overly-large waves don't bring down our main servers. We definitely want to be scraped. To your point: I think you're simply mistaken that they offer a good scraper-detection solution, but I'm all ears.