2 ms·
Wow, that's nuts. What sort of site was this? Mine don't get anything near this level of attention!
by AussieWog93 2mo ago
Wow, that's nuts. What sort of site was this? Mine don't get anything near this level of attention!
- loloquwowndueo 2mo agoJust a personal blog, and I have to admit it had a misconfiguration that was my fault - nonexistent urls would redirect to the index. Still, dumb scraper bots would 1) not realize there was a redirect loop and kept trying every possible url, and 2) they were doing so aggressively enough that it brought the site down a couple of times. A well-behaved crawler would have slowed down and noticed the loop, never had issues with crawlers for years until the advent of ai.