3 ms·

Meta's crawler made 11M requests to my site in 30 days

57 points by speckx 10mo ago

laborcontract 10mo ago
Obviously horrendous but why isn’t this person monitoring his site? Also, why do people use vercel nowadays? I’m sure there are reasons, but I moved over to railway (you can insert alternative provider here) and I no long f* around trying to fix page load time due to cold starts, I have predictable pricing, and my sites on railway are fast so much faster. Plus, if cost is a factor, railway offers serverless. It’s not as shiny as vercel, but nextjs works perfectly on it. It astounds me that vercel has positioned themselves as a sanctuary city for normies and yet, the city is littered with landmines and booby traps.
- spiderfarmer 10mo ago
  Don’t underestimate the amount of people who don’t care how their companies money is spent.
danpalmer 10mo ago
Meta shouldn't be doing this, they need to be more careful, but... I used to work on a site with basic caching, a big web framework, every page dynamic, and 3 frontend webservers plus a database primary and replica. Super basic infra, and a bill close to this user. We would never have noticed 3 to 4 requests per second like this. And we weren't being that smart about it, we were rendering every page not serving from cache (we mostly cached DB results). We were also conscious of not accidentally building SEO bot traps that would cause them to go around in loops, not because of the traffic generated, but because it was bad for SEO! This just strikes me as bad engineering on both sides. Yes Meta is the one with the big budgets and they should sort this out, but also you can't pay 10-100x for your infra and get annoyed when you have a big bill. On the web people and bots are going to make requests and you just have to design for that.
spiderfarmer 10mo ago
I have the same problem. I have 6M URL’s per domain. 8 different domains. 80% of search traffic is long tail. If I don’t block, 95% of my resources will be spent on feeding bots. I had to block all “official” AI useragents and entire countries like Singapore and China. But there are so many unofficial bots which spread their work over dozes of IP addresses that it seems impossible to block on the reverse proxy level. How do you block those?
- decremental 10mo ago
  [dead]
- kjok 10mo ago
  Block based on cookies (i.e., set a cookie on the browser and check on the server whether it exists).
  - direwolf20 10mo ago
    This project implements a variety of similar JSless checks, such as image loading https://github.com/WeebDataHoarder/go-away https://github.com/WeebDataHoarder/go-away
  - bigbadfeline 10mo ago
    That helps, but the big bot farms use clients that support cookies, we need to add more defenses on top of them.
- JasonADrury 10mo ago
  >If I don’t block, 95% of my resources will be spent on feeding bots. Okay, but why should you care? Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. You can easily serve 100k+ requests per second on servers that costs less than $100/mo. It really shouldn't be worth your time to do anything about the bots unless you're doing something unusual.
  - 10mo ago
blell 10mo ago
Crazy to me that someone would run a website where you pay for every request you receive, instead of a fixed monthly rate. It’s an obvious recipe for disaster - crossing the wrong guy would cost you dearly. Or just a crawler running amok.
JasonADrury 10mo ago
That's like 4 requests per second, hardly seems excessive at all. We're not on dial-up anymore.
- reassess_blind 10mo ago
  You’re not serious, right?
  - JasonADrury 10mo ago
    I am. Modern computers and network connections are so fast that this amounts to literally nothing. It's standard internet background noise and it's really not a problem.
  - spiderfarmer 10mo ago
    [flagged]
    - direwolf20 10mo ago
      I don't have a car. I don't need one because trains exist. My website can also handle 4 requests per second.
    - JasonADrury 10mo ago
      > He probably thinks the internet works on static 5k html pages, while the norm is 100kb, dynamically generated pages. I just work on web stuff that people actually use. It's 2026, thousands of requests per second is nothing. You'll probably be fine even with stock apache2 and some janky php scripts. A single gbit line will serve a 100kB page thousand times a second without issues. Dynamically generated pages you can't easily serve at rates in excess of tens of thousands of requests per second from commodity hardware are extremely rare.
      - danpalmer 10mo ago
        For most web apps, bandwidth won't be the issue, it'll still be I/O bound, or maybe CPU bound.
deleted 10mo ago
[deleted]
direwolf20 10mo ago
So sue Meta. Denial of service is a crime.

57 points by speckx 10mo ago

Obviously horrendous but why isn’t this person monitoring his site? Also, why do people use vercel nowadays? I’m sure there are reasons, but I moved over to railway (you can insert alternative provider here) and I no long f* around trying to fix page load time due to cold starts, I have predictable pricing, and my sites on railway are fast so much faster. Plus, if cost is a factor, railway offers serverless. It’s not as shiny as vercel, but nextjs works perfectly on it. It astounds me that vercel has positioned themselves as a sanctuary city for normies and yet, the city is littered with landmines and booby traps.

spiderfarmer 10mo ago
Don’t underestimate the amount of people who don’t care how their companies money is spent.

danpalmer 10mo ago

Meta shouldn't be doing this, they need to be more careful, but... I used to work on a site with basic caching, a big web framework, every page dynamic, and 3 frontend webservers plus a database primary and replica. Super basic infra, and a bill close to this user. We would never have noticed 3 to 4 requests per second like this. And we weren't being that smart about it, we were rendering every page not serving from cache (we mostly cached DB results). We were also conscious of not accidentally building SEO bot traps that would cause them to go around in loops, not because of the traffic generated, but because it was bad for SEO! This just strikes me as bad engineering on both sides. Yes Meta is the one with the big budgets and they should sort this out, but also you can't pay 10-100x for your infra and get annoyed when you have a big bill. On the web people and bots are going to make requests and you just have to design for that.

spiderfarmer 10mo ago

I have the same problem. I have 6M URL’s per domain. 8 different domains. 80% of search traffic is long tail. If I don’t block, 95% of my resources will be spent on feeding bots. I had to block all “official” AI useragents and entire countries like Singapore and China. But there are so many unofficial bots which spread their work over dozes of IP addresses that it seems impossible to block on the reverse proxy level. How do you block those?

decremental 10mo ago
[dead]
kjok 10mo ago
Block based on cookies (i.e., set a cookie on the browser and check on the server whether it exists).
- direwolf20 10mo ago
  This project implements a variety of similar JSless checks, such as image loading https://github.com/WeebDataHoarder/go-away https://github.com/WeebDataHoarder/go-away
- bigbadfeline 10mo ago
  That helps, but the big bot farms use clients that support cookies, we need to add more defenses on top of them.
JasonADrury 10mo ago
>If I don’t block, 95% of my resources will be spent on feeding bots. Okay, but why should you care? Resource usage for a regular website that isn't doing some heavy dynamic stuff or video streaming tends to be rather negligible. You can easily serve 100k+ requests per second on servers that costs less than $100/mo. It really shouldn't be worth your time to do anything about the bots unless you're doing something unusual.
- 10mo ago

blell 10mo ago

Crazy to me that someone would run a website where you pay for every request you receive, instead of a fixed monthly rate. It’s an obvious recipe for disaster - crossing the wrong guy would cost you dearly. Or just a crawler running amok.

JasonADrury 10mo ago

That's like 4 requests per second, hardly seems excessive at all. We're not on dial-up anymore.

reassess_blind 10mo ago
You’re not serious, right?
- JasonADrury 10mo ago
  I am. Modern computers and network connections are so fast that this amounts to literally nothing. It's standard internet background noise and it's really not a problem.
- spiderfarmer 10mo ago
  [flagged]
  - direwolf20 10mo ago
    I don't have a car. I don't need one because trains exist. My website can also handle 4 requests per second.
  - JasonADrury 10mo ago
    > He probably thinks the internet works on static 5k html pages, while the norm is 100kb, dynamically generated pages. I just work on web stuff that people actually use. It's 2026, thousands of requests per second is nothing. You'll probably be fine even with stock apache2 and some janky php scripts. A single gbit line will serve a 100kB page thousand times a second without issues. Dynamically generated pages you can't easily serve at rates in excess of tens of thousands of requests per second from commodity hardware are extremely rare.
    - danpalmer 10mo ago
      For most web apps, bandwidth won't be the issue, it'll still be I/O bound, or maybe CPU bound.

deleted 10mo ago

[deleted]

direwolf20 10mo ago

So sue Meta. Denial of service is a crime.