9 ms·

Amazon bares its computers

124 points by DLay 13y ago

venus 13y ago
> 1.5 million requests a second, and holds trillions of “objects” My god. I used to think I was a bit of a rock star, with (rails) web sites exceeding 100 requests/second on my resume. 1.5 million requests per second ... I cannot even really imagine that.
- codex 13y ago
  These get requests are likely sharded among a large number of machines. Viewed that way, it's much more manageable. Want to service more requests? Just add more machines. 1.5M relational queries on the other hand--that would be impressive.
- StavrosK 13y ago
  Over 15,000 machines, it's roughly the same thing!
- lukeschlather 13y ago
  It all depends on how many servers you have, as well as what sort of databases/database complexity. For a laughably trivial site, a single host could do a lot of requests per second without too much effort. For a normal Rails app running on a handful of hosts, 100 requests per second is nothing to scoff at, especially if you're on your own.
- ddeck 13y ago
  These are spread over an enormous number of machines. OT, but for truly impressive single machine performance, check out the LMAX FX trading platform, which can handle 6 million orders per second on a single core on commodity hardware.....in Java. [1] [1] http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html
jsnk 13y ago
>“All of our engineers are focused” on reducing the costs of computing If this is so, how is it that a startup like Digital Ocean that's only 2 years old find a way to provide $5/month plan that's running on SSD while m1.small on EC2 which is slower than DO costing me around $40/month? At this rate, I should probably buy a server myself and run it at home. I like AWS and its ecosystem, but I am tired of Amazon continuing to provide shitty service for a ridiculous cost.
- walshemj 13y ago
  MM yes I was at silicon milk about today ad everyone was on AWS - which is slightly worying. I think that a lot of the start ups around silicon roundabout woudl be a bit lost if they had to pivot and run their own kit. the CTO says the VC says we have to switch to self hosting we have 4 racks of dual core 1u 's arriving - anyone know how to put them together :-)
  - dmourati 13y ago
    Just find a sysadmin with > 5 years experience, ie, pre-cloud. We all know how it's done.
- beachstartup 13y ago
  right, amazon's engineers are focused on reducing the costs ... for amazon.
  - gtaylor 13y ago
    They cut prices for customers many times a year. Seems like we get 2-3 EC2 price cuts every year, and S3 continues to get cheaper. They are one of the few vendors we use that regularly passes on savings to us.
    - imbriaco 13y ago
      Compare their cost reductions with the decrease in computing costs and you'll very rapidly discover that they don't actually pass on savings to customers. Quite the opposite.
rottyguy 13y ago
Not really meant to be a slam, but startling to me that even after all these years, Amazon still has a PE of 1330+.
- 3825 13y ago
  I assume this means the market expects Amazon to make more money (a lot of money) in the future.
  - rottyguy 13y ago
    Yep, I get that. But they IPO'd in 1997. Again, not to detract, and in fact I'm amazed by how they've been able to compete both on the Brick & Mortar stage and Digital Services stage. I've just never seen a company given such love/leniency for so long by Wall St.
    - adventured 13y ago
      On Wall Street, the fake love often turns to a company being out of favor quickly (with both directions being equally irrational, in the overvaluing and then undervaluing). Indeed, the less fundamentals you have propping you up, the more likely this is to eventually happen, and happen hard. That exact scenario has played out numerous times with Netflix, for example. Amazon's valuation is almost purely an emotional basis, as all future returns are being pulled forward to the extent that they're overvalued and the calculations about their future earnings expectations are correct. Is Amazon worth $50 billion? $100b? $168b? That's an emotional judgment today, as they have no profits and little history of profitability, and if you want to speculate on tomorrow... well if their valuation keeps bubbling up, they'll soon catch Walmart, which has $17 billion in annual profit (meaning it may only take... 20 years or so for Amazon to fully realize that market value via profitability). Amazon is going to be an excellent short at the end of this stock market bubble, they'll lose half their value on the down swing (my opinion). However keep in mind, this is not a company that has always been loved by Wall Street. For most of its life, it has been hated by Wall Street. If you run an average price from early 2003 through the end of 2008, the stock was essentially flat lined for five years. And obviously it was viewed with near universal disgust after the dotcom bubble imploded, and the stock cratered to the $5's after 9/11. Amazon.Toast and all that.
kevinchen 13y ago
> Google publishes some technical papers about its big computers, but does not talk about things like the semiconductors it has developed internally to handle computing at this scale. Google designs its own ICs now?
- jws 13y ago
  I think I recall them making custom 10gbit Ethernet switch fabrics.
  - kijiki 13y ago
    They used merchant silicon in their custom switches. Modern switch ASICs are 2+ billion transistor devices with lots of fancy custom logic for SRAMs and TCAMs and analog magic for PHYs. Even Google can't afford to do that by itself.
    - twic 13y ago
      Pfft. They can just map-reduce over each bit in the frame.
- seiji 13y ago
  For over ten years. They're also the world's largest computer manufacturer (for internal data center hardware). Software people tend to think chips and hardware design are magic skills known by only three revered people in the world. It's just another track of school you could have chosen. ("Google writes its own software now?")
colmmacc 13y ago
Full-disclosure: I'm an engineer at Amazon Web Services. "Mr. Hamilton later said privately that Amazon had developed original statistical methods to limit damage from catastrophic failures." I don't think it's what James is referring to (as it's just the tip of the iceberg), but at Re:Invent we also released Infima - a library for service-level fault isolation and fault tolerance using Route 53 for endpoint discovery; https://github.com/awslabs/route53-infima https://github.com/awslabs/route53-infima The techniques in Infima give some insight into how we think about multi-tenancy and availability, as well as provide a framework for building other services on top of EC2, Route 53 and so on.
- simonebrunozzi 13y ago
  This one looks great! Any talk/video available that describe it in more details?
mwilcox 13y ago
"You've reached your limit of 10 free articles a month." ugh.
- medell 13y ago
  Open in a private browsing window.
  - natrius 13y ago
    Or, you know, pay for the product. I don't understand the logic behind complaining about paying $3.75 a week for a product that you use so frequently. It should be embarrassing.
    - seiji 13y ago
      Their pricing model grates at me. You want to charge more to read your content in mobile apps in addition to normal web pages?
    - jlgreco 13y ago
      10 articles a month isn't exactly frequently. At the low end, you are suggesting that people throw more than a dollar per article at them. You shouldn't find it surprising that many people will baulk at that.
kraemate 13y ago
Anyone have the link to the original presentation? Particularly intrigued by the statistical fault tolerance.
- dbarlett 13y ago
  Not the whole presentation, but some slides and better notes: http://gigaom.com/2013/11/15/how-amazon-is-building-substations-laying-fiber-and-generally-doing-everything-to-keep-cloud-costs-down/ http://gigaom.com/2013/11/15/how-amazon-is-building-substati...
anan0s 13y ago
I'm a bit skeptical on the fact that such companies do not disclose their custom infrastructure design. So their value is both in hardware optimizations as well as software ones ? being a grad student myself, I'm wondering if we'll ever see any specific technical aspects on infrastructures that could drive systems research and provide real problems to be attacked...
- XorNot 13y ago
  I'm skeptical that there's anything that innovative going on behind the veil. I suspect if you had access you would look at a whole bunch of evolutionary improvements and probably trace them back to which papers you've seen it sort of mentioned in. Probably not as much custom silicon as the article implies either, simply because on the scale of things Amazon is not a semiconductor company and Intel pretty unambiguously own the cutting edge there.
- wmf 13y ago
  Facebook reveals their infrastructure through the Open Compute Project since it can't compete against their core business.
b0b0b0b 13y ago
James Hamilton's blog is not to be missed: http://perspectives.mvdirona.com/ http://perspectives.mvdirona.com/
locusm 13y ago
A Dell account exec told me last week that Dell had built and implemented the Sydney AWS data center - any truth to that statement?
- nl 13y ago
  They are in Equinix SY3 (Sydney)[1]. Pretty sure Dell doesn't do DC design for Equinix. I heard some stories about the truck loads of servers going in when they were rolling it out, but I don't remember the vendor. [1] http://www.arnnet.com.au/article/442144/equinix_expands_its_sydney_sy3_datacentre_28_photos_/ http://www.arnnet.com.au/article/442144/equinix_expands_its_...
11thEarlOfMar 13y ago
What is rather mind boggling is that Google, Amazon, Facebook, Microsoft, Yahoo all have multiple, huge data center installations and, they are all profitable companies. It does not seem to be a zero-sum game yet for market share, and the growth in the world's appetite for storage and access is still not fully served. Can they all wind up 'winning' with profitable operations founded on web infrastructure? How much storage does one Internet citizen occupy, anyway?
- adventured 13y ago
  "How much storage does one Internet citizen occupy, anyway?" That number chases toward infinite. Its growth may slow down, as we reach the limit of how many 'actions' a person can reasonable take that generates content, but it'll never stop growing just on the basis of the unlimited desire for higher quality experiences (mixed with a general moore's law ride that makes it all financially reasonable). And just when you think storage demand is going to slow down, we'll jump to some kind of virtual reality life recording, that makes it possible for others to walk around in your daily life as though they're actually there, and storage demand will explode another magnitude. You can remove Facebook and Yahoo from that list though. Their operations are funded by their core businesses and they don't compete in the AWS-style cloud segment. That is, it doesn't make sense to discuss Facebook in regard to whether it can win or make money, compared to AWS or Azure. So is there room for Google, AWS, Azure, Rackspace, IBM etc? Yep. It's a future hundred billion dollar plus business (AWS will probably be at $10b+ in a few years). No reason there can't be three or four big players in such a large market.
badmadrad 13y ago
The price will bottom out eventually and Amazon knows this. What we are really seeing is an infrastructure arms race and that is why AWS is costly. They are not building AWS for the individual user long term. They are building it for the enterprise. Eventually as bandwidth and computing power increases and cost decreases it may be feasible to run your personal blog out of your house or use a VPS on Digital Ocean. However, as we become a global community you will have to think about your application on that scale to be successful and that's where AWS can find longevity.
dschiptsov 13y ago
VmWare is a much greater waste.
eitally 13y ago
Interesting that they focus on all the infrastructure they design and build/commission themselves. They are also a big customer of my company (http://www.newisys.com/Products/products.shtml http://www.newisys.com/Products/products.shtml). Coincidentally, they call out Facebook at the bottom of the article, in context of the Open Compute Project. Facebook had airflow and overheating problems with their intended OCP design and our Newisys engineers helped with some design adjustments to resolve those, too. Now Facebook is buying cheap Taiwanese-made (Quanta & Compal) OCP-design servers and shipping them to us (http://sanmina.com/ http://sanmina.com/) for racking, systems test, and shipping to their DC in Lulea, Sweden (https://www.facebook.com/LuleaDataCenter https://www.facebook.com/LuleaDataCenter). So yes, these guys all do lots of tough engineering work, but it's not in isolation and not without assistance from industry veterans whose careers it has been to ensure properly functioning data centers and networks.
bluedino 13y ago
>> Mr. Hamilton also said Amazon was not interested in participating in industry standards discussions, which could publicize its many technical advances. What are the side effects of this in a few years, when all of the high-end stuff has been designed 4 different ways by 4 different companies? What happens when Dell and HP's server businesses are shells of their former selves? When Cisco is but a memory as their biggest customers have began designing solutions in-house? You'll have to go with the cloud. You just won't be able to get the best of the best equipment without doing so.

124 points by DLay 13y ago

venus 13y ago

> 1.5 million requests a second, and holds trillions of “objects” My god. I used to think I was a bit of a rock star, with (rails) web sites exceeding 100 requests/second on my resume. 1.5 million requests per second ... I cannot even really imagine that.

codex 13y ago
These get requests are likely sharded among a large number of machines. Viewed that way, it's much more manageable. Want to service more requests? Just add more machines. 1.5M relational queries on the other hand--that would be impressive.
StavrosK 13y ago
Over 15,000 machines, it's roughly the same thing!
lukeschlather 13y ago
It all depends on how many servers you have, as well as what sort of databases/database complexity. For a laughably trivial site, a single host could do a lot of requests per second without too much effort. For a normal Rails app running on a handful of hosts, 100 requests per second is nothing to scoff at, especially if you're on your own.
ddeck 13y ago
These are spread over an enormous number of machines. OT, but for truly impressive single machine performance, check out the LMAX FX trading platform, which can handle 6 million orders per second on a single core on commodity hardware.....in Java. [1] [1] http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html

jsnk 13y ago

>“All of our engineers are focused” on reducing the costs of computing If this is so, how is it that a startup like Digital Ocean that's only 2 years old find a way to provide $5/month plan that's running on SSD while m1.small on EC2 which is slower than DO costing me around $40/month? At this rate, I should probably buy a server myself and run it at home. I like AWS and its ecosystem, but I am tired of Amazon continuing to provide shitty service for a ridiculous cost.

walshemj 13y ago
MM yes I was at silicon milk about today ad everyone was on AWS - which is slightly worying. I think that a lot of the start ups around silicon roundabout woudl be a bit lost if they had to pivot and run their own kit. the CTO says the VC says we have to switch to self hosting we have 4 racks of dual core 1u 's arriving - anyone know how to put them together :-)
- dmourati 13y ago
  Just find a sysadmin with > 5 years experience, ie, pre-cloud. We all know how it's done.
beachstartup 13y ago
right, amazon's engineers are focused on reducing the costs ... for amazon.
- gtaylor 13y ago
  They cut prices for customers many times a year. Seems like we get 2-3 EC2 price cuts every year, and S3 continues to get cheaper. They are one of the few vendors we use that regularly passes on savings to us.
  - imbriaco 13y ago
    Compare their cost reductions with the decrease in computing costs and you'll very rapidly discover that they don't actually pass on savings to customers. Quite the opposite.

rottyguy 13y ago

Not really meant to be a slam, but startling to me that even after all these years, Amazon still has a PE of 1330+.

3825 13y ago
I assume this means the market expects Amazon to make more money (a lot of money) in the future.
- rottyguy 13y ago
  Yep, I get that. But they IPO'd in 1997. Again, not to detract, and in fact I'm amazed by how they've been able to compete both on the Brick & Mortar stage and Digital Services stage. I've just never seen a company given such love/leniency for so long by Wall St.
  - adventured 13y ago
    On Wall Street, the fake love often turns to a company being out of favor quickly (with both directions being equally irrational, in the overvaluing and then undervaluing). Indeed, the less fundamentals you have propping you up, the more likely this is to eventually happen, and happen hard. That exact scenario has played out numerous times with Netflix, for example. Amazon's valuation is almost purely an emotional basis, as all future returns are being pulled forward to the extent that they're overvalued and the calculations about their future earnings expectations are correct. Is Amazon worth $50 billion? $100b? $168b? That's an emotional judgment today, as they have no profits and little history of profitability, and if you want to speculate on tomorrow... well if their valuation keeps bubbling up, they'll soon catch Walmart, which has $17 billion in annual profit (meaning it may only take... 20 years or so for Amazon to fully realize that market value via profitability). Amazon is going to be an excellent short at the end of this stock market bubble, they'll lose half their value on the down swing (my opinion). However keep in mind, this is not a company that has always been loved by Wall Street. For most of its life, it has been hated by Wall Street. If you run an average price from early 2003 through the end of 2008, the stock was essentially flat lined for five years. And obviously it was viewed with near universal disgust after the dotcom bubble imploded, and the stock cratered to the $5's after 9/11. Amazon.Toast and all that.

kevinchen 13y ago

> Google publishes some technical papers about its big computers, but does not talk about things like the semiconductors it has developed internally to handle computing at this scale. Google designs its own ICs now?

jws 13y ago
I think I recall them making custom 10gbit Ethernet switch fabrics.
- kijiki 13y ago
  They used merchant silicon in their custom switches. Modern switch ASICs are 2+ billion transistor devices with lots of fancy custom logic for SRAMs and TCAMs and analog magic for PHYs. Even Google can't afford to do that by itself.
  - twic 13y ago
    Pfft. They can just map-reduce over each bit in the frame.
seiji 13y ago
For over ten years. They're also the world's largest computer manufacturer (for internal data center hardware). Software people tend to think chips and hardware design are magic skills known by only three revered people in the world. It's just another track of school you could have chosen. ("Google writes its own software now?")

colmmacc 13y ago

Full-disclosure: I'm an engineer at Amazon Web Services. "Mr. Hamilton later said privately that Amazon had developed original statistical methods to limit damage from catastrophic failures." I don't think it's what James is referring to (as it's just the tip of the iceberg), but at Re:Invent we also released Infima - a library for service-level fault isolation and fault tolerance using Route 53 for endpoint discovery; https://github.com/awslabs/route53-infima https://github.com/awslabs/route53-infima The techniques in Infima give some insight into how we think about multi-tenancy and availability, as well as provide a framework for building other services on top of EC2, Route 53 and so on.

simonebrunozzi 13y ago
This one looks great! Any talk/video available that describe it in more details?

mwilcox 13y ago

"You've reached your limit of 10 free articles a month." ugh.

medell 13y ago
Open in a private browsing window.
- natrius 13y ago
  Or, you know, pay for the product. I don't understand the logic behind complaining about paying $3.75 a week for a product that you use so frequently. It should be embarrassing.
  - seiji 13y ago
    Their pricing model grates at me. You want to charge more to read your content in mobile apps in addition to normal web pages?
  - jlgreco 13y ago
    10 articles a month isn't exactly frequently. At the low end, you are suggesting that people throw more than a dollar per article at them. You shouldn't find it surprising that many people will baulk at that.

kraemate 13y ago

Anyone have the link to the original presentation? Particularly intrigued by the statistical fault tolerance.

dbarlett 13y ago
Not the whole presentation, but some slides and better notes: http://gigaom.com/2013/11/15/how-amazon-is-building-substations-laying-fiber-and-generally-doing-everything-to-keep-cloud-costs-down/ http://gigaom.com/2013/11/15/how-amazon-is-building-substati...

anan0s 13y ago

I'm a bit skeptical on the fact that such companies do not disclose their custom infrastructure design. So their value is both in hardware optimizations as well as software ones ? being a grad student myself, I'm wondering if we'll ever see any specific technical aspects on infrastructures that could drive systems research and provide real problems to be attacked...

XorNot 13y ago
I'm skeptical that there's anything that innovative going on behind the veil. I suspect if you had access you would look at a whole bunch of evolutionary improvements and probably trace them back to which papers you've seen it sort of mentioned in. Probably not as much custom silicon as the article implies either, simply because on the scale of things Amazon is not a semiconductor company and Intel pretty unambiguously own the cutting edge there.
wmf 13y ago
Facebook reveals their infrastructure through the Open Compute Project since it can't compete against their core business.

b0b0b0b 13y ago

James Hamilton's blog is not to be missed: http://perspectives.mvdirona.com/ http://perspectives.mvdirona.com/

locusm 13y ago

A Dell account exec told me last week that Dell had built and implemented the Sydney AWS data center - any truth to that statement?

nl 13y ago
They are in Equinix SY3 (Sydney)[1]. Pretty sure Dell doesn't do DC design for Equinix. I heard some stories about the truck loads of servers going in when they were rolling it out, but I don't remember the vendor. [1] http://www.arnnet.com.au/article/442144/equinix_expands_its_sydney_sy3_datacentre_28_photos_/ http://www.arnnet.com.au/article/442144/equinix_expands_its_...

11thEarlOfMar 13y ago

What is rather mind boggling is that Google, Amazon, Facebook, Microsoft, Yahoo all have multiple, huge data center installations and, they are all profitable companies. It does not seem to be a zero-sum game yet for market share, and the growth in the world's appetite for storage and access is still not fully served. Can they all wind up 'winning' with profitable operations founded on web infrastructure? How much storage does one Internet citizen occupy, anyway?

adventured 13y ago
"How much storage does one Internet citizen occupy, anyway?" That number chases toward infinite. Its growth may slow down, as we reach the limit of how many 'actions' a person can reasonable take that generates content, but it'll never stop growing just on the basis of the unlimited desire for higher quality experiences (mixed with a general moore's law ride that makes it all financially reasonable). And just when you think storage demand is going to slow down, we'll jump to some kind of virtual reality life recording, that makes it possible for others to walk around in your daily life as though they're actually there, and storage demand will explode another magnitude. You can remove Facebook and Yahoo from that list though. Their operations are funded by their core businesses and they don't compete in the AWS-style cloud segment. That is, it doesn't make sense to discuss Facebook in regard to whether it can win or make money, compared to AWS or Azure. So is there room for Google, AWS, Azure, Rackspace, IBM etc? Yep. It's a future hundred billion dollar plus business (AWS will probably be at $10b+ in a few years). No reason there can't be three or four big players in such a large market.

badmadrad 13y ago

The price will bottom out eventually and Amazon knows this. What we are really seeing is an infrastructure arms race and that is why AWS is costly. They are not building AWS for the individual user long term. They are building it for the enterprise. Eventually as bandwidth and computing power increases and cost decreases it may be feasible to run your personal blog out of your house or use a VPS on Digital Ocean. However, as we become a global community you will have to think about your application on that scale to be successful and that's where AWS can find longevity.

dschiptsov 13y ago

VmWare is a much greater waste.

eitally 13y ago

Interesting that they focus on all the infrastructure they design and build/commission themselves. They are also a big customer of my company (http://www.newisys.com/Products/products.shtml http://www.newisys.com/Products/products.shtml). Coincidentally, they call out Facebook at the bottom of the article, in context of the Open Compute Project. Facebook had airflow and overheating problems with their intended OCP design and our Newisys engineers helped with some design adjustments to resolve those, too. Now Facebook is buying cheap Taiwanese-made (Quanta & Compal) OCP-design servers and shipping them to us (http://sanmina.com/ http://sanmina.com/) for racking, systems test, and shipping to their DC in Lulea, Sweden (https://www.facebook.com/LuleaDataCenter https://www.facebook.com/LuleaDataCenter). So yes, these guys all do lots of tough engineering work, but it's not in isolation and not without assistance from industry veterans whose careers it has been to ensure properly functioning data centers and networks.

bluedino 13y ago

>> Mr. Hamilton also said Amazon was not interested in participating in industry standards discussions, which could publicize its many technical advances. What are the side effects of this in a few years, when all of the high-end stuff has been designed 4 different ways by 4 different companies? What happens when Dell and HP's server businesses are shells of their former selves? When Cisco is but a memory as their biggest customers have began designing solutions in-house? You'll have to go with the cloud. You just won't be able to get the best of the best equipment without doing so.