2 ms·
Meta shouldn't be doing this, they need to be more careful, but... I used to work on a site with basic caching, a big web framework, every page dynamic, and 3
by danpalmer 9mo ago
Meta shouldn't be doing this, they need to be more careful, but...
I used to work on a site with basic caching, a big web framework, every page dynamic, and 3 frontend webservers plus a database primary and replica. Super basic infra, and a bill close to this user.
We would never have noticed 3 to 4 requests per second like this. And we weren't being that smart about it, we were rendering every page not serving from cache (we mostly cached DB results). We were also conscious of not accidentally building SEO bot traps that would cause them to go around in loops, not because of the traffic generated, but because it was bad for SEO!
This just strikes me as bad engineering on both sides. Yes Meta is the one with the big budgets and they should sort this out, but also you can't pay 10-100x for your infra and get annoyed when you have a big bill. On the web people and bots are going to make requests and you just have to design for that.