3 ms·
I believe you. But at the same time it's insane to me we have these standards that aren't upheld on one end or the other. I'm fulling willing to acknowledge tha
by tstrimple 2y ago
I believe you. But at the same time it's insane to me we have these standards that aren't upheld on one end or the other. I'm fulling willing to acknowledge that Anthropic is the problem here. But at some level, and at some point, don't you lose the ability to say the site is down because it's too popular? I'm totally open to the arguments that the traffic overloaded the Cloudflare limits or whatever. But the frequency of outages caused by not just traffic at Anthropic levels, but at random HN post levels are enough to take down websites. I guess this is just my cognitive dissonance between being told on HN "you don't need to scale! You're not Facebook!" and "OMG our went down because we have a few thousand concurrent visitors!"
- stetrain 2y agoI don't see where iFixit said this caused an outage.
- tracker1 2y agoI don't think the site itself went down, so much as the scraping itself was excessive and beyond this, the presumed use of said scraping is against the TOS and copyrights of the site itself. On a smaller site, it can absolutely be brutal. I worked on a site where literally 2/3 of requests were bots and a lot of those were dynamically constructed pages that are/were harder to cache. In the end, switching to a supplemental search database helped significantly, but it doesn't mean that an aggressive bot is an okay thing.