4 ms·
From the tweet: > Hey @AnthropicAI: I get you're hungry for data. Claude is really smart! But do you really need to hit our servers a million times in 24 hours?
by rkwz 2y ago
From the tweet:
> Hey @AnthropicAI: I get you're hungry for data. Claude is really smart! But do you really need to hit our servers a million times in 24 hours?
You're not only taking our content without paying, you're tying up our devops resources. Not cool.
Assuming they're literally getting hit "a million times in 24 hours", wouldn't it still cost bandwidth? Also, perhaps iFixit doesn't have such traffic in their past so they didn't bother optimzing prematurely.
I feel the blame mostly falls on Anthropic not being considerate when they scrape other people's content.
- tstrimple 2y ago> I feel the blame mostly falls on Anthropic not being considerate when they scrape other people's content. That's certainly a fair assumption. I haven't looked into it in any detail. Is IFixit sending appropriate cache headers or is Anthropic just ignoring them? I don't know. But at some point, your caching strategy and your ability to respond to spikes in demand is your own problem. We live in reality, and in reality you can't really control who is hitting your website when. Sure, it would be nice if they respected your cache control policies, but what are you doing when you know they won't? I'll refer back to my previous statement. It's totally understandable when someone who didn't expect ever HN levels of attentions to be overwhelmed. But if you're a site as large and respected as IFixit, you've got to do better than the "unwashed masses".