3 ms·
I had to put my search engine behind Cloudflare to deal with this. Like the volume grew to about 10x the traffic I saw sitting at the front page of Hacker News
by marginalia_nu 5y ago
I had to put my search engine behind Cloudflare to deal with this. Like the volume grew to about 10x the traffic I saw sitting at the front page of Hacker News for a full week.
- marginalia_nu 5y agoThis is the rate of rejected HTTP requests I'm seeing at this point: https://www.marginalia.nu/junk/spam.png https://www.marginalia.nu/junk/spam.png Real search traffic is about half that.
- m-i-l 5y agoThanks V. I'm seeing a similar number of problem search requests (although nowhere near as many real search requests:-), so it is probably the same "SEO practitioners" running the same "scraping footprints" against different search engines around the same time. I was kind-of hoping that somewhere in this discussion there would be an "And the answer to your problem is...", but I suppose it is a very specific problem which only a search engine would encounter. I think the Cloudflare solution you have is probably the best to block the requests as early as possible. The reverse proxy config[0] I've got seems to be mostly holding out for now though. [0] https://github.com/searchmysite/searchmysite.net/issues/55 https://github.com/searchmysite/searchmysite.net/issues/55
- marginalia_nu 5y agoIf they're from the same outfit I've had problems with I really am at a loss as to what, other than Cloudflare, is a good solution. I got like 4-5 requests per second at worst. Seems to be a botnet, I entered a few of the source IPs into my browser and got like login screens to enterprise routers and so on.