3 ms·
Doesn't it get blocked by IP rate limiting? Nobody should be able to make so many calls to your site that it results in 10x higher bill on your side. Even if n
by raunaksingwi 17d ago
Doesn't it get blocked by IP rate limiting? Nobody should be able to make so many calls to your site that it results in 10x higher bill on your side.
Even if not a crawler, it's basically a DoS attach.
More over, why does muse have to crawl your urls so many times?
- Marciok 17d agoVercel "lets it sleep through" since they make money on these calls. I have no idea why it needs to crawl so many times; I guess that Meta is probably a mess right now, with lots of researchers training many models at the same time.