3 ms·

I've been afraid to use S3 like this because S3 requests sometimes fail and the client needs to retry after a short delay. S3 libraries do retries under the ho

by agwa 14y ago

I've been afraid to use S3 like this because S3 requests sometimes fail and the client needs to retry after a short delay. S3 libraries do retries under the hood, but web browsers don't - instead the user sees an error message or a broken page. This probably wouldn't happen much on a low-traffic site, but a site getting as many hits as this guy's probably has problems for a non-trivial number of users.

blowski 14y ago
Do you have any data on how much is 'non-trivial'? (Not disbelieving you, I'm wondering whether it's trivial for my use case.)
- agwa 14y ago
  Here's a HN comment I bookmarked about this: https://news.ycombinator.com/item?id=4976893 https://news.ycombinator.com/item?id=4976893 That guy was reporting a 2% error rate from one S3 server, but there has been additional discussion on that thread which suggests that error rate is an aberration. Personally, I have no hard data, just the anecdote that when I started developing with S3 I wasn't using retries and that became a problem after approximately a few thousand queries. Unfortunately I've never seen Amazon say what error rate to expect, just that you should be sure to retry on error. I think I'll start logging how often I need to retry so I can get some hard data. Keep in mind that when you're hosting a site on S3, you have to multiply the error rate by the number of resources you're serving from S3 to get the probability an individual user will have problems. www.ismytwitterpasswordsecure.com serves 4 resources from S3. Assuming a 0.1% error rate, 400 out of the 100,000 users would have had problems. As another commenter has suggested, you can serve from Cloudfront instead of directly from S3. That will significantly reduce the number of hits which are made to S3.
untog 14y ago
I had read that, and was expecting that (for a silly idea like this I don't really care), but I never received any feedback from anyone that received an error.
andrewmunsell 14y ago
Well, you could always use CloudFront as well, which would prevent these errors from ever appearing to your website's visitors.

by agwa 14y ago

blowski 14y ago

Do you have any data on how much is 'non-trivial'? (Not disbelieving you, I'm wondering whether it's trivial for my use case.)

agwa 14y ago
Here's a HN comment I bookmarked about this: https://news.ycombinator.com/item?id=4976893 https://news.ycombinator.com/item?id=4976893 That guy was reporting a 2% error rate from one S3 server, but there has been additional discussion on that thread which suggests that error rate is an aberration. Personally, I have no hard data, just the anecdote that when I started developing with S3 I wasn't using retries and that became a problem after approximately a few thousand queries. Unfortunately I've never seen Amazon say what error rate to expect, just that you should be sure to retry on error. I think I'll start logging how often I need to retry so I can get some hard data. Keep in mind that when you're hosting a site on S3, you have to multiply the error rate by the number of resources you're serving from S3 to get the probability an individual user will have problems. www.ismytwitterpasswordsecure.com serves 4 resources from S3. Assuming a 0.1% error rate, 400 out of the 100,000 users would have had problems. As another commenter has suggested, you can serve from Cloudfront instead of directly from S3. That will significantly reduce the number of hits which are made to S3.

untog 14y ago

I had read that, and was expecting that (for a silly idea like this I don't really care), but I never received any feedback from anyone that received an error.

andrewmunsell 14y ago

Well, you could always use CloudFront as well, which would prevent these errors from ever appearing to your website's visitors.