6 ms·

Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3

61 points by werner 16y ago

kennymeyers 16y ago
I can't believe this hasn't received more comments. I find this absolutely astonishing and promising. The future is mighty bright.
riobard 16y ago
I guess we have to thank Netflix for this? :D
- polynomial 16y ago
  Netflix Cloud Architect Adrian Cockcroft explains: http://blip.tv/file/4252897 http://blip.tv/file/4252897 slides here: http://www.slideshare.net/adrianco/netflix-on-cloud-combined-slides-for-dev-and-ops http://www.slideshare.net/adrianco/netflix-on-cloud-combined...
zacman85 16y ago
It is hard for me to even imagine what the world was like before Amazon Web Services, despite having only used them for half of a year. I truly hope Amazon is making ungodly sums of money with this increasingly powerful platform. Why Google is laser-focused on social and local, yet letting Amazon eat their lunch in this space is beyond me. I know Google has AppEngine and Storage, but they seem to be lacking the full-featured, cohesive collection of cloud services that Amazon has so clearly invested themselves in. I truly wonder if we will wake up five years from now and the vast majority of the web will be running on Amazon infrastructure.
- ssmoot 16y ago
  No. It's much more cost effective to run your own infrastructure. It's always struck me as irresponsible for reedit to spend the sums they do for example. You could easily spend half that and get four times the hardware (in addition to an admin to manage it). Some businesses just don't want to deal with it period. I get that. But at some point it's almost irresponsible. Like sending an employee to starbucks for everyone in the office a few times a day because you don't want the hassle of setting up with a coffee service. ;-)
  - 9oliYQjP 16y ago
    What would you estimate Reddit's costs are now? What would their costs be if they had a full-time (decent) admin and their own hardware in a data centre?
    - lnguyen 16y ago
      Actually you probably should add in a team of admins unless you assume a single person is capable of 24/7 service and handle the possibility of multiple large-scale issues simultaneously. Also factor in expertise (not just familiarity) in all areas of infrastructure.
      - Uchikoma 16y ago
        You need the admins either way, AWS does not help with that. Most problems from experience are with the application (Database, Memory, Locking, ...) not with the infrastructure (Load balancer, Linux, ...). [Edit]: The problems Reddit, Digg, Foursquare had were problems with application code and application infrastructure (MongoDB, Cassandra, RabbitMQ, Redis, ...). AWS admins would not - and have not - helped with that. So you need your own admins, AWS does not spare you those.
  - Uchikoma 16y ago
    This is what's amazing me too. AWS is so much more expensive, even if factoring TCO in. It's good for very fast scaling and experimentation tough. I've did some very rough calculations in a blog post some time ago http://codemonkeyism.com/dark-side-virtualized-servers-cloud/ http://codemonkeyism.com/dark-side-virtualized-servers-cloud... and AWS seems to be a least 2x more expensive.
  - zacman85 16y ago
    I understand you are right in the context of today. A few years from now, however, I would imagine that gains in the efficiency of the business model as well as competitive pressures will drive prices down below the costs of running your own infrastructure.
  - 16y ago
- patio11 16y ago
  I think Google considers their infrastructure and expertise with web-scale data processing to be a durable source of competitive advantage. You need to be substantially as good as Google to do web-scale crawling and analysis to build a general purpose search engine. Google makes billions upon billions of dollars from having that available. Are they going to make billions upon billions of dollars with a huge competitive moat and 30% gross margins by publishing their competitive advantage as OSS and getting into hosting? Probably not. Which is not to say Google isn't a friend of OSS and transparency -- indeed, they are the world's most reliable fans of OSSing the competitive advantages of companies which are not Google.
  - potatolicious 16y ago
    > "Are they going to make billions upon billions of dollars with a huge competitive moat and 30% gross margins by publishing their competitive advantage as OSS and getting into hosting? Probably not." But are they going to make billions upon billions of dollars when their competitors have instant access to a service that levels the playing field? [edit] Okay, not quite level, but even with Amazon collecting their margin, it dramatically closes the gap in highly scaled computing, to the point where if I were Google I would not rely on cost advantage as a long-term strategy. Competing on price is rarely a game you want to get into, it tends to turn into a race to the bottom more often than not.
  - brown9-2 16y ago
    What does OSS have to do with this? Amazon isn't releasing the source for their infrastructure services.
Yeroc 16y ago
How long before the AWS division is spun off as it's own independent company?
- cperciva 16y ago
  Technically AWS is a separate company: Amazon Web Services LLC. Presumably Amazon.com is the only shareholder, though. I doubt AWS will go public, though -- it's too central to Amazon's existence. Amazon has never really been about books.
dotcoma 16y ago
great. unless it's Wikileaks documents...
madmath 16y ago
"But I just want to serve 5 Terabytes!" I like that they increased the limit, not 5x, not 100x, but 1000x. It clearly sends the message that they are the leaders in cloud storage.

61 points by werner 16y ago

I can't believe this hasn't received more comments. I find this absolutely astonishing and promising. The future is mighty bright.

riobard 16y ago

I guess we have to thank Netflix for this? :D

polynomial 16y ago
Netflix Cloud Architect Adrian Cockcroft explains: http://blip.tv/file/4252897 http://blip.tv/file/4252897 slides here: http://www.slideshare.net/adrianco/netflix-on-cloud-combined-slides-for-dev-and-ops http://www.slideshare.net/adrianco/netflix-on-cloud-combined...

zacman85 16y ago

It is hard for me to even imagine what the world was like before Amazon Web Services, despite having only used them for half of a year. I truly hope Amazon is making ungodly sums of money with this increasingly powerful platform. Why Google is laser-focused on social and local, yet letting Amazon eat their lunch in this space is beyond me. I know Google has AppEngine and Storage, but they seem to be lacking the full-featured, cohesive collection of cloud services that Amazon has so clearly invested themselves in. I truly wonder if we will wake up five years from now and the vast majority of the web will be running on Amazon infrastructure.

ssmoot 16y ago
No. It's much more cost effective to run your own infrastructure. It's always struck me as irresponsible for reedit to spend the sums they do for example. You could easily spend half that and get four times the hardware (in addition to an admin to manage it). Some businesses just don't want to deal with it period. I get that. But at some point it's almost irresponsible. Like sending an employee to starbucks for everyone in the office a few times a day because you don't want the hassle of setting up with a coffee service. ;-)
- 9oliYQjP 16y ago
  What would you estimate Reddit's costs are now? What would their costs be if they had a full-time (decent) admin and their own hardware in a data centre?
  - lnguyen 16y ago
    Actually you probably should add in a team of admins unless you assume a single person is capable of 24/7 service and handle the possibility of multiple large-scale issues simultaneously. Also factor in expertise (not just familiarity) in all areas of infrastructure.
    - Uchikoma 16y ago
      You need the admins either way, AWS does not help with that. Most problems from experience are with the application (Database, Memory, Locking, ...) not with the infrastructure (Load balancer, Linux, ...). [Edit]: The problems Reddit, Digg, Foursquare had were problems with application code and application infrastructure (MongoDB, Cassandra, RabbitMQ, Redis, ...). AWS admins would not - and have not - helped with that. So you need your own admins, AWS does not spare you those.
- Uchikoma 16y ago
  This is what's amazing me too. AWS is so much more expensive, even if factoring TCO in. It's good for very fast scaling and experimentation tough. I've did some very rough calculations in a blog post some time ago http://codemonkeyism.com/dark-side-virtualized-servers-cloud/ http://codemonkeyism.com/dark-side-virtualized-servers-cloud... and AWS seems to be a least 2x more expensive.
- zacman85 16y ago
  I understand you are right in the context of today. A few years from now, however, I would imagine that gains in the efficiency of the business model as well as competitive pressures will drive prices down below the costs of running your own infrastructure.
- 16y ago
patio11 16y ago
I think Google considers their infrastructure and expertise with web-scale data processing to be a durable source of competitive advantage. You need to be substantially as good as Google to do web-scale crawling and analysis to build a general purpose search engine. Google makes billions upon billions of dollars from having that available. Are they going to make billions upon billions of dollars with a huge competitive moat and 30% gross margins by publishing their competitive advantage as OSS and getting into hosting? Probably not. Which is not to say Google isn't a friend of OSS and transparency -- indeed, they are the world's most reliable fans of OSSing the competitive advantages of companies which are not Google.
- potatolicious 16y ago
  > "Are they going to make billions upon billions of dollars with a huge competitive moat and 30% gross margins by publishing their competitive advantage as OSS and getting into hosting? Probably not." But are they going to make billions upon billions of dollars when their competitors have instant access to a service that levels the playing field? [edit] Okay, not quite level, but even with Amazon collecting their margin, it dramatically closes the gap in highly scaled computing, to the point where if I were Google I would not rely on cost advantage as a long-term strategy. Competing on price is rarely a game you want to get into, it tends to turn into a race to the bottom more often than not.
- brown9-2 16y ago
  What does OSS have to do with this? Amazon isn't releasing the source for their infrastructure services.

Yeroc 16y ago

How long before the AWS division is spun off as it's own independent company?

cperciva 16y ago
Technically AWS is a separate company: Amazon Web Services LLC. Presumably Amazon.com is the only shareholder, though. I doubt AWS will go public, though -- it's too central to Amazon's existence. Amazon has never really been about books.

dotcoma 16y ago

great. unless it's Wikileaks documents...

madmath 16y ago

"But I just want to serve 5 Terabytes!" I like that they increased the limit, not 5x, not 100x, but 1000x. It clearly sends the message that they are the leaders in cloud storage.