4 ms·

Amazon S3 - Bigger and Busier Than Ever

44 points by jeffbarr 16y ago

saurik 16y ago
I'm pretty certain that I store almost a billion objects into S3 every month (yes: the requests for that cost a lot)... does that make me a noticeable percentage of S3's object count? Seriously now?
- jimmybot 16y ago
  Don't keep us in suspense, what are you storing?
  - saurik 16y ago
    I wish it was interesting, or even sane... one day my website got like a million more users /that day/ and every database mechanism I had to do some of the user statistics I was keeping totally failed: the website ground to a halt. So, rather than decide what I really wanted to keep, I frantically developed a little library that stored all of that data into S3 as a giant key/value store, with my own little makeshift inverted indexes. Needless to say, this was stupid, and when I next checked my S3 bills I noticed I was doing a billion puts per month (which costs like $10,000). Lukily, this has only been a few months, and I'm now fixing the situation during a giant cost-cotti g endeavor... but yeah: omg that was stupid. (typed on my iPhone and therefore a little shortened: hacker news comments suck on the iPhone)
    - jimmybot 16y ago
      Interesting, thanks for sharing. I'm a packrat myself, but I guess something like S3 where you actively pay for storage and transport shows you very clearly the cost of being a packrat--there may be value in what you keep around, but it's not free to keep around, and it's not free to move it to storage.
marcamillion 16y ago
Is it just me, or does this post leave you a bit confused. What does 292 billion objects mean? I know an object is a file stored in a bucket. But rly? If Amazon wants to convince developers to use their platform they have to speak in terms that ordinary people can understand. Tell us how much money developers have saved since they switched - how many individual apps are using S3, etc.
sandipagr 16y ago
wow the growth is phenomenal! Everything is indeed going to be in the cloud
- mikeknoop 16y ago
  It is kind of funny, right? I mean, our servers are already "the cloud". So it's a really cloud within a cloud.
ashleyw 16y ago
Off-topic: I've always wondered who 'Jeff' is on the AWS blog…I like to think it's Bezos, but is it?
- axomhacker 16y ago
  It's Jeff Barr, Lead AWS Evangelist. http://news.ycombinator.com/user?id=jeffbarr http://news.ycombinator.com/user?id=jeffbarr (the OP). http://www.jeff-barr.com/?page_id=670 http://www.jeff-barr.com/?page_id=670 edit: Barr, not Bar :(
  - jeffbarr 16y ago
    Yes, that's me, but with two r's. When I started writing the AWS blog in late 2004 I wasn't sure how personal I should or could be, and didn't know if it was ok to brand it with my full name. I had adopted "-- Jeff;" as my signature a long time ago, when one of my friends had "++md;"
- wiredfool 16y ago
  Nope, It's Jeff Barr. http://www.jeff-barr.com/ http://www.jeff-barr.com/ [[edit]] that'll teach me not to refresh before commenting.
Groxx 16y ago
Looking at it from a bit more of a critical view than their graph would probably appreciate, that means that from the end of 2007 on they've done a little better than doubling yearly. That's it? They're an enormous company and seem to me to be by far the most-mentioned storage option. I'm surprised they're not a heck of a lot higher, especially given that a number of very popular cloud services sit right on top of S3 and store many gigabytes and hammer it with requests. Oh well. If they maintain that growth, they'll own everything soon enough. I don't mean to imply yearly-doubling is bad in any way, just that I would've expected more by now.
- axomhacker 16y ago
  Sure, depending on the definition of little better: Yearly growth (# of objects): 2.9 > 14 > 40 > 102 > 262 Yearly growth (as a ratio): 4.8 > 2.85 > 2.55 > 2.56 Also - this is just the number of objects stored (not put in ever) Obviously because storage costs, folks have purging solutions. It'd be interesting to see a similar trend for requests/sec.
- turbodog 16y ago
  Is cloud storage the factor that is driving companies to AWS? I'm sure that's huge for NetFlix, but they are an outlier. My bet would be on EC2 usage as being a better growth metric.
  - Groxx 16y ago
    Well, to pick a high-profile AWS-user here: all of Dropbox runs on S3, and apparently a bit of EC2, last I saw. Not that I think they account for a truly significant amount of Amazon's traffic, but they're not small-time either.

44 points by jeffbarr 16y ago

I'm pretty certain that I store almost a billion objects into S3 every month (yes: the requests for that cost a lot)... does that make me a noticeable percentage of S3's object count? Seriously now?

jimmybot 16y ago
Don't keep us in suspense, what are you storing?
- saurik 16y ago
  I wish it was interesting, or even sane... one day my website got like a million more users /that day/ and every database mechanism I had to do some of the user statistics I was keeping totally failed: the website ground to a halt. So, rather than decide what I really wanted to keep, I frantically developed a little library that stored all of that data into S3 as a giant key/value store, with my own little makeshift inverted indexes. Needless to say, this was stupid, and when I next checked my S3 bills I noticed I was doing a billion puts per month (which costs like $10,000). Lukily, this has only been a few months, and I'm now fixing the situation during a giant cost-cotti g endeavor... but yeah: omg that was stupid. (typed on my iPhone and therefore a little shortened: hacker news comments suck on the iPhone)
  - jimmybot 16y ago
    Interesting, thanks for sharing. I'm a packrat myself, but I guess something like S3 where you actively pay for storage and transport shows you very clearly the cost of being a packrat--there may be value in what you keep around, but it's not free to keep around, and it's not free to move it to storage.

marcamillion 16y ago

Is it just me, or does this post leave you a bit confused. What does 292 billion objects mean? I know an object is a file stored in a bucket. But rly? If Amazon wants to convince developers to use their platform they have to speak in terms that ordinary people can understand. Tell us how much money developers have saved since they switched - how many individual apps are using S3, etc.

sandipagr 16y ago

wow the growth is phenomenal! Everything is indeed going to be in the cloud

mikeknoop 16y ago
It is kind of funny, right? I mean, our servers are already "the cloud". So it's a really cloud within a cloud.

ashleyw 16y ago

Off-topic: I've always wondered who 'Jeff' is on the AWS blog…I like to think it's Bezos, but is it?

axomhacker 16y ago
It's Jeff Barr, Lead AWS Evangelist. http://news.ycombinator.com/user?id=jeffbarr http://news.ycombinator.com/user?id=jeffbarr (the OP). http://www.jeff-barr.com/?page_id=670 http://www.jeff-barr.com/?page_id=670 edit: Barr, not Bar :(
- jeffbarr 16y ago
  Yes, that's me, but with two r's. When I started writing the AWS blog in late 2004 I wasn't sure how personal I should or could be, and didn't know if it was ok to brand it with my full name. I had adopted "-- Jeff;" as my signature a long time ago, when one of my friends had "++md;"
wiredfool 16y ago
Nope, It's Jeff Barr. http://www.jeff-barr.com/ http://www.jeff-barr.com/ [[edit]] that'll teach me not to refresh before commenting.

Groxx 16y ago

Looking at it from a bit more of a critical view than their graph would probably appreciate, that means that from the end of 2007 on they've done a little better than doubling yearly. That's it? They're an enormous company and seem to me to be by far the most-mentioned storage option. I'm surprised they're not a heck of a lot higher, especially given that a number of very popular cloud services sit right on top of S3 and store many gigabytes and hammer it with requests. Oh well. If they maintain that growth, they'll own everything soon enough. I don't mean to imply yearly-doubling is bad in any way, just that I would've expected more by now.

axomhacker 16y ago
Sure, depending on the definition of little better: Yearly growth (# of objects): 2.9 > 14 > 40 > 102 > 262 Yearly growth (as a ratio): 4.8 > 2.85 > 2.55 > 2.56 Also - this is just the number of objects stored (not put in ever) Obviously because storage costs, folks have purging solutions. It'd be interesting to see a similar trend for requests/sec.
turbodog 16y ago
Is cloud storage the factor that is driving companies to AWS? I'm sure that's huge for NetFlix, but they are an outlier. My bet would be on EC2 usage as being a better growth metric.
- Groxx 16y ago
  Well, to pick a high-profile AWS-user here: all of Dropbox runs on S3, and apparently a bit of EC2, last I saw. Not that I think they account for a truly significant amount of Amazon's traffic, but they're not small-time either.