7 ms·

Amazon is building substations, laying fiber and keeping cloud costs down

81 points by antr 13y ago

protomyth 13y ago
"Back in the day, Hamilton used to lobby for just having one or two SKUs from a server vendor in order to minimize complexity, but times have changed. Once you master the process, going straight to server manufacturers with custom designs can lop 30 percent off the price right away, not to mention the improved performance and faster turnaround time." So, they get the server manufacturer to do the work instead of building it themselves. I guess the scale goes: low volume: buy -> build own -> custom outsource :high volume I wonder how long each design lasts before they move on.
happycube 13y ago
Everything at Amazon is about creating scale. Prime exists to get shipping volumes up far enough to pull things like trucking items daily from Vegas to Southern California, directly into the USPS system at the sorting facilities - and eventually direct deliveries. The other thing - which lead to AWS, FBA, etc, was that each Amazon department had to make a globally usable API, even for (then!) inside facing stuff. Combining those and you get a giant. Hopefully Bezos will stick around for a while yet - I can't see Amazon continuing to not maximize profits in exchange for volume very long after he's gone.
- pyre 13y ago
  What happens to the kingdom once the "good king" passes away... Edit: I was angling more at the fact that Amazon has been very dominant, and that's not going away soon. So if it starts getting bad, it's not like there will be something just as good overnight. Look at how long it took to erode MS's dominance from the 90's.
  - pstuart 13y ago
    Empires come, and empires go.
  - evv 13y ago
    The economy thrives because of the increased competition..
  - aquadrop 13y ago
    And ironically, MS is the one which can prevent Amazon from doing bad things. Since it's a very large company with a lot of resources and they invest heavily in Azure. Google stands near with its compute engine. And there's bunch of smaller companies ready to compete. I think this market isn't in danger of monopolization.
  - WesternStar 13y ago
    Honestly, if I'm Amazon or Google I'm in that business not really to make money. Hell, I'll run the same business at a loss frankly because I have to build that business anyway to run the business I'm already in. I need the infrastructure to run a scalable website if my competitors and others want to pay me to do that ,great.
antr 13y ago
I might be naïve with the following comment: I would love to see AWS moving a bit further down the customer chain, providing one-click installations to non-savvy customers (I'm thinking on-click wordpress, magento, docker, etc.). Given the recent acquisition of MediaTemple by SlowDaddy I'm confident many GridServer customers would of moved from MT to AWS.
- count 13y ago
  Lots of companies already do this. AWS provides CloudFormation templates that setup whole environments in one (or so) clicks. Bitnami and others provide pre-built, operational Wordpress and other stuff. There's a ton of stuff available.
- atonse 13y ago
  The beauty of AWS is that any company can build exactly what you describe, on top of AWS.
- jgreen10 13y ago
  https://aws.amazon.com/marketplace https://aws.amazon.com/marketplace tip: search for wordpress, magento The biggest drawback of AWS is that you miss features not because they don't have them, but because they have so many of them. :)
- kbar13 13y ago
  https://aws.amazon.com/opsworks/ https://aws.amazon.com/opsworks/
- beambot 13y ago
  Check out BitNami. This is exactly what they do. (No affiliation.)
alexkus 13y ago
> Hamilton didn’t talk a lot about AWS’s custom-built storage, but he did share one tidbit. The densest storage servers you can buy commercially today come from Quanta, and a rack full of them would weigh in at about three-quarters of a ton. “We have a far denser design — it is more than a ton,” Hamilton said. Interesting. For reference; a Backblaze pod is 4U and weighs 145 pounds, so 10 of them in a standard 42U rack would be 1450 pounds = 0.715 (short) tons (or ~648kg).
- sparkman55 13y ago
  For many colo centers, there are both weight and power limitations on a per-rack (or per-cabinet / per-row) basis. One wouldn't want a ton of disks crashing through the (often raised) floor! For compute clusters, power is often the limiting factor; for storage clusters, weight is often the limiting factor. If you put these clusters in a general-purpose colocation facility, you'll end up with half-full racks. Density on a per-rack-unit basis is not necessarily important for most of us. Of course, if you're Amazon's size, you build your own facilities, and you can make sure to support the weight and power requirements of whatever custom hardware you can dream up!
- deleted 13y ago
  [deleted]
- rgbrenner 13y ago
  quanta is 60 3.5" hdds in 4u (compared with 45 for backblaze)
malandrew 13y ago
Their comments about storage cluster density are interesting. I'm still shocked that hard drive design basically hasn't changed at all. What I'd like to see is hard-drives that don't need to be screwed in and don't need sleds. Instead they are machined with grooves on the tops, bottoms and sides to be "slide in, slide out" and have the grooves be multipurpose in the sense that they are designed for airflow. i.e. the top of one hard drive mates with the bottom of an identical hard drive to create channels through which air can be forced through and cool both drives maximally, simultaneously. Basically take the level of machining Apple puts into designing the the Macbook Pros and apply it to hard drives. If designed correctly, you should be able to pack them together, side by side in a way that they lock (like a jigsaw puzzle or the Heineken World Bottle[0]) in a stable "lattice" of hard drives that only permit sliding out to one side and permit efficient cooling. Screws and sleds are so yesteryear. [0] http://www.fastcodesign.com/1672684/heineken-s-lost-plan-to-build-houses-out-of-beer-bottles http://www.fastcodesign.com/1672684/heineken-s-lost-plan-to-... [1] I hope that HN comments like these constitute prior art in case someone decides to go out and do this and patent the idea.
- jemfinch 13y ago
  Your idea sounds heavy. A lot of storage deployments are limited (surprisingly) not by space or power, but by the weight that can be supported on the raised floors common in POPs, so something that increases (storage) density while simultaneously increasing (mass) density would be a non-starter.
- deleted 13y ago
  [deleted]
- sigstoat 13y ago
  since the tops are thin sheet metal, and the bottoms are circuit boards, i don't think we'll see mating surfaces there. but they could probably make some side grooves, and little cutouts for spring loaded latches to catch on. add some cutouts on the front that something could grab onto, and you might be able to avoid the need for the sled and screws, without building up an entire latticework out of drives. there probably needs to be structure to hold the back plane(s) and line them up with the drives, anyways.
taway2012 13y ago
Pretty interesting nugget there. I've always thought Amazon cloud services were overpriced. But I thought maybe the software engineering labor is expensive. The comment that "software engineering costs rounds down to zero" was extra information I didn't have that strengthens my gut feeling. Just my two cents. I don't expect to change anybody's mind. :)
- Patrick_Devine 13y ago
  Everyone gets a raise! Seriously. I'm not sure how well compensated AWS engineers are, however if the price engineering is just noise, why not pay the best in the industry?
  - thinkling 13y ago
    Have you looked at what Amazon stock has done over the past few years?
    - Patrick_Devine 13y ago
      Sure, but realistically what's the upside? Maybe your stock doubles if Amazon gets to be as big as Microsoft, but that's unlikely given how razor thin the margins are. Sure, it could happen, but if Amazon is spending several billion on data centers and warehouses every year, that's likely to make a dent in the stock price.
  - donavanm 13y ago
    Assume an SDE/SE/NE costs $125,000 per year on average. Multiply that by 10,000 employees to reach $1.25B per year. A 3% raise costs you $37M dollars. Not an insubstantial amount of profit to give up.
aquadrop 13y ago
I wonder how come Amazon didn't mop up the floor with his competitors (e.g. DigitalOcean, Rackspace) price-wise if AWS infrastructure is so optimized? Of course, they have more features, but that's all kinda software-related, and they say software doesn't cost them that much.
- eloff 13y ago
  In high-end servers they blow the competition away on price. I did a comparison recently and they came out over 4x cheaper than Rackspace for two 256GB of RAM servers + heavy bandwidth.
  - vidarh 13y ago
    Blowing Rackspace away on price is trivial - they're one of the most expensive hosting options I've ever looked at.
  - adventured 13y ago
    This isn't even remotely accurate. I could give you a dozen examples, but one will suffice. From a hosting provider I've used for many years, WebNX. Here are two example configurations from them: E5-1650v2, 64gb ram, 4x 450gb SAS 15,000 rpm enterprise drives and hardware raid 10 with battery backup, 20tb of bandwidth - $289 / month, no setup fee Dual 2690v2, 256gb ram, 6x 600GB 15k sas raid 10 w/ 2x SSD Cache - $950 / month, no setup fee The bandwidth alone would cost you $2,300 at Amazon. And 20tb is nothing these days, countless quality providers offer 100mbps standard with $150 machines. The AWS cr1.8xlarge with just 20tb of bandwidth is $5,134 per month. A one year locked in contract gets it down to $2,815 ($7,200 up front) - three times more expensive. And there are far cheaper providers than WebNX. AWS is almost interesting, if you never need any bandwidth.
    - eloff 13y ago
      That's also an over-simplification. For example at AWS you can get a fully bidirectional 10gbps link, regardless of what other tenants are doing on the network. Nobody else that I'm aware of can match that. And you'll quickly find that 20TB of bandwidth at those network speeds are nothing, you can cross into petabytes without breaking a sweat (1 petabyte at 50% of bandwidth 24x7 for a month for just one server.) At that kind of network volume bandwidth costs dominate everything (about $65K for an petabyte of bandwidth at Amazon.) I challenge you to beat that anywhere. Also if you have bursty traffic patterns, AWS comes out a lot cheaper because it's designed around quickly scaling up and down capacity. In the end it depends on matching a provider to your particular needs because they vary wildy in strengths and weaknesses and costs.
polysing 13y ago
This is all very interesting and impressive, but why doesn't anyone talk about how much pollution and environmental damage this is causing? It's terrifying to think about how much fossil fuel will be burned to keep these resources running. How many servers and cables they will be throwing away every year. How much packaging trash is being generated by Amazon every day. Do these inconvenient by-products simply disappear? No, they show up in your landfills, and in your oceans, and they get "exported" (dumped) in China and Africa. The only metrics people are looking at is revenue, profits, growth, bandwidth and storage. But how about we start measuring pollution, waste, and environmental destruction for every company, right alongside the traditional metrics? It's great when the stock price doubles. But does anyone think about the fact that this probably also means the amount of plastic trash in our oceans, and poisonous gas in our atmosphere doubles?
- Joyfield 13y ago
  You could stop using the Internet and really make a change.
- wmf 13y ago
  The environmental impact of cloud computing is hotly debated. To the extent that it's replacing "OldOps" kind of infrastructure it's probably more efficient. OTOH the Jevons Paradox predicts that as the cloud gets cheaper it will be used for increasingly higher volumes of lower-value tasks that previously would not have been done at all. Ultimately I favor internalizing the environmental costs and letting Amazon deal with them how they will.

81 points by antr 13y ago

"Back in the day, Hamilton used to lobby for just having one or two SKUs from a server vendor in order to minimize complexity, but times have changed. Once you master the process, going straight to server manufacturers with custom designs can lop 30 percent off the price right away, not to mention the improved performance and faster turnaround time." So, they get the server manufacturer to do the work instead of building it themselves. I guess the scale goes: low volume: buy -> build own -> custom outsource :high volume I wonder how long each design lasts before they move on.

happycube 13y ago

Everything at Amazon is about creating scale. Prime exists to get shipping volumes up far enough to pull things like trucking items daily from Vegas to Southern California, directly into the USPS system at the sorting facilities - and eventually direct deliveries. The other thing - which lead to AWS, FBA, etc, was that each Amazon department had to make a globally usable API, even for (then!) inside facing stuff. Combining those and you get a giant. Hopefully Bezos will stick around for a while yet - I can't see Amazon continuing to not maximize profits in exchange for volume very long after he's gone.

pyre 13y ago
What happens to the kingdom once the "good king" passes away... Edit: I was angling more at the fact that Amazon has been very dominant, and that's not going away soon. So if it starts getting bad, it's not like there will be something just as good overnight. Look at how long it took to erode MS's dominance from the 90's.
- pstuart 13y ago
  Empires come, and empires go.
- evv 13y ago
  The economy thrives because of the increased competition..
- aquadrop 13y ago
  And ironically, MS is the one which can prevent Amazon from doing bad things. Since it's a very large company with a lot of resources and they invest heavily in Azure. Google stands near with its compute engine. And there's bunch of smaller companies ready to compete. I think this market isn't in danger of monopolization.
- WesternStar 13y ago
  Honestly, if I'm Amazon or Google I'm in that business not really to make money. Hell, I'll run the same business at a loss frankly because I have to build that business anyway to run the business I'm already in. I need the infrastructure to run a scalable website if my competitors and others want to pay me to do that ,great.

antr 13y ago

I might be naïve with the following comment: I would love to see AWS moving a bit further down the customer chain, providing one-click installations to non-savvy customers (I'm thinking on-click wordpress, magento, docker, etc.). Given the recent acquisition of MediaTemple by SlowDaddy I'm confident many GridServer customers would of moved from MT to AWS.

count 13y ago
Lots of companies already do this. AWS provides CloudFormation templates that setup whole environments in one (or so) clicks. Bitnami and others provide pre-built, operational Wordpress and other stuff. There's a ton of stuff available.
atonse 13y ago
The beauty of AWS is that any company can build exactly what you describe, on top of AWS.
jgreen10 13y ago
https://aws.amazon.com/marketplace https://aws.amazon.com/marketplace tip: search for wordpress, magento The biggest drawback of AWS is that you miss features not because they don't have them, but because they have so many of them. :)
kbar13 13y ago
https://aws.amazon.com/opsworks/ https://aws.amazon.com/opsworks/
beambot 13y ago
Check out BitNami. This is exactly what they do. (No affiliation.)

alexkus 13y ago

> Hamilton didn’t talk a lot about AWS’s custom-built storage, but he did share one tidbit. The densest storage servers you can buy commercially today come from Quanta, and a rack full of them would weigh in at about three-quarters of a ton. “We have a far denser design — it is more than a ton,” Hamilton said. Interesting. For reference; a Backblaze pod is 4U and weighs 145 pounds, so 10 of them in a standard 42U rack would be 1450 pounds = 0.715 (short) tons (or ~648kg).

sparkman55 13y ago
For many colo centers, there are both weight and power limitations on a per-rack (or per-cabinet / per-row) basis. One wouldn't want a ton of disks crashing through the (often raised) floor! For compute clusters, power is often the limiting factor; for storage clusters, weight is often the limiting factor. If you put these clusters in a general-purpose colocation facility, you'll end up with half-full racks. Density on a per-rack-unit basis is not necessarily important for most of us. Of course, if you're Amazon's size, you build your own facilities, and you can make sure to support the weight and power requirements of whatever custom hardware you can dream up!
deleted 13y ago
[deleted]
rgbrenner 13y ago
quanta is 60 3.5" hdds in 4u (compared with 45 for backblaze)

malandrew 13y ago

Their comments about storage cluster density are interesting. I'm still shocked that hard drive design basically hasn't changed at all. What I'd like to see is hard-drives that don't need to be screwed in and don't need sleds. Instead they are machined with grooves on the tops, bottoms and sides to be "slide in, slide out" and have the grooves be multipurpose in the sense that they are designed for airflow. i.e. the top of one hard drive mates with the bottom of an identical hard drive to create channels through which air can be forced through and cool both drives maximally, simultaneously. Basically take the level of machining Apple puts into designing the the Macbook Pros and apply it to hard drives. If designed correctly, you should be able to pack them together, side by side in a way that they lock (like a jigsaw puzzle or the Heineken World Bottle[0]) in a stable "lattice" of hard drives that only permit sliding out to one side and permit efficient cooling. Screws and sleds are so yesteryear. [0] http://www.fastcodesign.com/1672684/heineken-s-lost-plan-to-build-houses-out-of-beer-bottles http://www.fastcodesign.com/1672684/heineken-s-lost-plan-to-... [1] I hope that HN comments like these constitute prior art in case someone decides to go out and do this and patent the idea.

jemfinch 13y ago
Your idea sounds heavy. A lot of storage deployments are limited (surprisingly) not by space or power, but by the weight that can be supported on the raised floors common in POPs, so something that increases (storage) density while simultaneously increasing (mass) density would be a non-starter.
deleted 13y ago
[deleted]
sigstoat 13y ago
since the tops are thin sheet metal, and the bottoms are circuit boards, i don't think we'll see mating surfaces there. but they could probably make some side grooves, and little cutouts for spring loaded latches to catch on. add some cutouts on the front that something could grab onto, and you might be able to avoid the need for the sled and screws, without building up an entire latticework out of drives. there probably needs to be structure to hold the back plane(s) and line them up with the drives, anyways.

taway2012 13y ago

Pretty interesting nugget there. I've always thought Amazon cloud services were overpriced. But I thought maybe the software engineering labor is expensive. The comment that "software engineering costs rounds down to zero" was extra information I didn't have that strengthens my gut feeling. Just my two cents. I don't expect to change anybody's mind. :)

Patrick_Devine 13y ago
Everyone gets a raise! Seriously. I'm not sure how well compensated AWS engineers are, however if the price engineering is just noise, why not pay the best in the industry?
- thinkling 13y ago
  Have you looked at what Amazon stock has done over the past few years?
  - Patrick_Devine 13y ago
    Sure, but realistically what's the upside? Maybe your stock doubles if Amazon gets to be as big as Microsoft, but that's unlikely given how razor thin the margins are. Sure, it could happen, but if Amazon is spending several billion on data centers and warehouses every year, that's likely to make a dent in the stock price.
- donavanm 13y ago
  Assume an SDE/SE/NE costs $125,000 per year on average. Multiply that by 10,000 employees to reach $1.25B per year. A 3% raise costs you $37M dollars. Not an insubstantial amount of profit to give up.

aquadrop 13y ago

I wonder how come Amazon didn't mop up the floor with his competitors (e.g. DigitalOcean, Rackspace) price-wise if AWS infrastructure is so optimized? Of course, they have more features, but that's all kinda software-related, and they say software doesn't cost them that much.

eloff 13y ago
In high-end servers they blow the competition away on price. I did a comparison recently and they came out over 4x cheaper than Rackspace for two 256GB of RAM servers + heavy bandwidth.
- vidarh 13y ago
  Blowing Rackspace away on price is trivial - they're one of the most expensive hosting options I've ever looked at.
- adventured 13y ago
  This isn't even remotely accurate. I could give you a dozen examples, but one will suffice. From a hosting provider I've used for many years, WebNX. Here are two example configurations from them: E5-1650v2, 64gb ram, 4x 450gb SAS 15,000 rpm enterprise drives and hardware raid 10 with battery backup, 20tb of bandwidth - $289 / month, no setup fee Dual 2690v2, 256gb ram, 6x 600GB 15k sas raid 10 w/ 2x SSD Cache - $950 / month, no setup fee The bandwidth alone would cost you $2,300 at Amazon. And 20tb is nothing these days, countless quality providers offer 100mbps standard with $150 machines. The AWS cr1.8xlarge with just 20tb of bandwidth is $5,134 per month. A one year locked in contract gets it down to $2,815 ($7,200 up front) - three times more expensive. And there are far cheaper providers than WebNX. AWS is almost interesting, if you never need any bandwidth.
  - eloff 13y ago
    That's also an over-simplification. For example at AWS you can get a fully bidirectional 10gbps link, regardless of what other tenants are doing on the network. Nobody else that I'm aware of can match that. And you'll quickly find that 20TB of bandwidth at those network speeds are nothing, you can cross into petabytes without breaking a sweat (1 petabyte at 50% of bandwidth 24x7 for a month for just one server.) At that kind of network volume bandwidth costs dominate everything (about $65K for an petabyte of bandwidth at Amazon.) I challenge you to beat that anywhere. Also if you have bursty traffic patterns, AWS comes out a lot cheaper because it's designed around quickly scaling up and down capacity. In the end it depends on matching a provider to your particular needs because they vary wildy in strengths and weaknesses and costs.

polysing 13y ago

This is all very interesting and impressive, but why doesn't anyone talk about how much pollution and environmental damage this is causing? It's terrifying to think about how much fossil fuel will be burned to keep these resources running. How many servers and cables they will be throwing away every year. How much packaging trash is being generated by Amazon every day. Do these inconvenient by-products simply disappear? No, they show up in your landfills, and in your oceans, and they get "exported" (dumped) in China and Africa. The only metrics people are looking at is revenue, profits, growth, bandwidth and storage. But how about we start measuring pollution, waste, and environmental destruction for every company, right alongside the traditional metrics? It's great when the stock price doubles. But does anyone think about the fact that this probably also means the amount of plastic trash in our oceans, and poisonous gas in our atmosphere doubles?

Joyfield 13y ago
You could stop using the Internet and really make a change.
wmf 13y ago
The environmental impact of cloud computing is hotly debated. To the extent that it's replacing "OldOps" kind of infrastructure it's probably more efficient. OTOH the Jevons Paradox predicts that as the cloud gets cheaper it will be used for increasingly higher volumes of lower-value tasks that previously would not have been done at all. Ultimately I favor internalizing the environmental costs and letting Amazon deal with them how they will.