3 ms·
That's still 518.400 requests per day. For static content. And it's a niche forum, so it's not exactly going to have millions of pages. Either there are indeed
by crote 11mo ago
That's still 518.400 requests per day. For static content. And it's a niche forum, so it's not exactly going to have millions of pages.
Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.
- n1xis10t 11mo agoImagine if all this scraping was going into a search engine with a massive index, or a bunch of smaller search engines that a meta-search engine could be made for. This’d be a lot more cool in that case