5 ms·

We used to build steel mills near cheap power, now that's where we put servers

63 points by luu 12y ago

krampian 12y ago
Actually that is still a pretty good location for steel mills, as well as other industries, like Bitcoin mining farms (perhaps those fall under datacenters...).
imglorp 12y ago
Being near a river wouldn't hurt for heat dissipation either.
- _dark_matter_ 12y ago
  Not great for the wildlife, however.
- wyldfire 12y ago
  Using a waterway as a heat sink for a server farm may not sound as bad as dumping waste, but it's bound to have an impact on the wildlife there.
  - TheGRS 12y ago
    I think the heat alone would disrupt a river ecosystem.
- thrownaway2424 12y ago
  Modern datacenters are not cooled that way, and some of them don't even have substantial makeup water inputs.
bsder 12y ago
Unfortunately, the baseline assumption isn't correct. Actually, we built steel mills near cheap transportation, not cheap power. Why? Because the main power source for primary steelmaking is coal (coke, actually), not electricity. And you need cheap transport for the power source (coal), raw materials (iron ore and such), and the finished steel. And cheap transportation, water, helps with getting rid of heat. Electricity is mostly used in secondary steelmaking where you are taking already existing steel, remelting it, and adjusting its properties. Now, aluminum production is very much tied to cheap electricity. I seem to remember that Iceland does a tremendous amount of aluminum refining because of the cheap geothermal power.
- twotwotwo 12y ago
  > Unfortunately, the baseline assumption isn't correct. To quibble with that quibble, though the steel mills thing is in the title, I wouldn't call it a "baseline assumption" of the longer discussion in the post: the post talks about how power affects datacenter costs, with one sentence at the end mentioning steel mills. He links the "steel mills" phrase to a page on a Microsoft datacenter in Quincy, WA. I don't know much about the history of that area, but I did read that another Microsoft datacenter was built in The Dalles, OR, which had very cheap power and historically was home to aluminum manufacturers that needed a lot of power for the electrolytic process involved: http://www.nytimes.com/2006/06/14/technology/14search.html?ei=&_r=0 http://www.nytimes.com/2006/06/14/technology/14search.html?e... mentions the datacenter and that aluminum smelters were once in the area http://en.wikipedia.org/wiki/Aluminium_smelting http://en.wikipedia.org/wiki/Aluminium_smelting talks more about the process and how it takes "prodigious amounts of electricity"
  - butchlugrod 12y ago
    Quincy, like The Dalles, is located near the Columbia River, and is an hour drive to the Grand Coulee, Chief Joseph, Wanapum, Rock Island, and Rocky Reach hydroelectric dams. Electricity in central Washington is very inexpensive. Since that article was written, Dell, Yahoo, Vantage, and Sabey have all constructed data centers in the Quincy area.
    - AllexS 12y ago
      Also note that a very large reason for constructing Rocky Reach Dam on the Columbia was to provide power for an Alcoa Aluminum plant that is still heavily dependent on cheap energy prices. Chelan, Douglas and Grant counties all have nearly the cheapest power rates the the country.
- tracker1 12y ago
  Beyond that, I'd say that low-latency abundant bandwidth availability is probably harder to find and bring into a site than power is these days. Not to mention effective cooling. Then again, there's some very large banking institutions with data centers here in the Phoenix area, so what do I know.
twotwotwo 12y ago
++luu for an interesting read. One thought on hardware acceleration: it may be kind of happening, piece by piece. AES-NI, carry-less multiply, and CRC32 instructions are kind of like smallish (in gate area) accelerators of operations that go much easier in hardware. AMD talked about a compression accelerator for their (now-delayed) A1100 ARM server chip. These are marketed as performance features, not as power-saving, but if you can do something in fewer clock cycles without a ton of area, you are often saving on power as well. It's also possible that some savings come through finding places where slower chips will do the job because RAM and/or disk and/or bandwidth is the limiter (storage, caching, maybe some proxies and scale-out DBs now running on faster chips). Intel has server Atoms (Avoton, Rangeley) and there are experiments with ARM servers (Baidu cloud storage tried them out: https://gigaom.com/2013/02/20/first-arm-based-servers-in-production-support-baidus-cloud-storage/ https://gigaom.com/2013/02/20/first-arm-based-servers-in-pro...). There's lots of work on ARMs and Atoms on the consumer side and some of the benefits may spill over to low-power servers. Don't know how widespread this will be, though; I've been rooting for the concept of servers on low-power chips for years and haven't seen much yet. You also pay for higher power dissipation with lower density. If you're Amazon or some huge provider, you can be talking about a lot of space in absolute terms. It's interesting to eyeball a latest-gen Facebook Open Compute server with high-wattage CPUs in it, and how much of its height seems to be due to those heatsinks and fans: http://images.anandtech.com/doci/9138/leopard-mob.jpg http://images.anandtech.com/doci/9138/leopard-mob.jpg SeaMicro sadly didn't succeed, but it's interesting they thought people wanted density enough to make it a key part of their marketing, and AMD thought enough of the idea to buy them: http://www.brightsideofnews.com/Data/2012_9_11/AMD-Expands-SeaMicro-Big-Data-Gets-a-High-Performance-Home/AMD_SeaMicro7_689.jpg http://www.brightsideofnews.com/Data/2012_9_11/AMD-Expands-S... Of course, companies using and selling blade servers and the popularity of <1U configs (e.g., SuperMicro's twin servers: http://www.supermicro.com/products/nfo/1UTwin.cfm http://www.supermicro.com/products/nfo/1UTwin.cfm, Microsoft's blade-y variation on the idea: http://www.anandtech.com/show/9138/open-compute-hardware-tried-and-tested/9 http://www.anandtech.com/show/9138/open-compute-hardware-tri..., even that Facebook Open Compute layout which fits 3 boxes in 2U) demonstrate some interest in density.
willholloway 12y ago
Solar panel prices have dropped to a point where we are about to enter an era of energy cost deflation. This is an incredible human achievement, and the climate and economic benefits will be enormous. The US was muddling its way out the 2008 crash repercussions, with incremental improvement in the economy. It wasn't until the oil-choke collar on the economy was removed that we saw a return to the low unemployment levels we are seeing today. Energy costs are a tax on everything we do, and we are in for a global energy tax cut. The American Southwest has tremendous solar resources, and could be the cheapest place to run data centers in the states if we could develop: 1. Lower cost electricity storage, compressed air, new batteries, or superconducting magnetic energy storage could do it. 2. Develop economical solar thermal air conditioning systems. If you can generate heat, you can generate cold. http://en.wikipedia.org/wiki/Solar_air_conditioning http://en.wikipedia.org/wiki/Solar_air_conditioning
- outworlder 12y ago
  Where can I source panels that cheap, if I'm not in the US. Genuine question, I'd really want to know.
  - toomuchtodo 12y ago
    China.
  - grecy 12y ago
    My parents in Australia just paid $10k AUD for 5KW fully installed including the inverter and "smart meter". (the government later reimbursed them $5k of that, though that's not really relevant)
- adventured 12y ago
  Your economic references are incorrect. 1) Unemployment levels are not low, yet. U6 is about 11%. It was 8% in 2007, and 7% in 2001. The labor force participation rate is at a 40 year low. Full-time employment has seen zero net gain in over a decade, despite a population increase of 30 million people. 2) US GDP growth has seen zero improvement. The GDP growth rate the last five years is lower than it has been in all of US history outside of recessions (the average rate of growth from the 1940s to 2007 was about 3.5% +/-). There has been no acceleration. Growth rates - 2010: 2.5%; 2011: 1.6%; 2012: 2.3%; 2013: 2.2%; 2014: 2.4% These growth rates required 0% interest rates, $8 trillion in new public debt, and trillions in Fed stimulus. This latest quarter required a -0.1% inflation adjuster (only the second time that has happened in 65 years...) to keep the economy from printing another negative GDP growth quarter. The oil and fossil fuels collar has not been removed from the US economy. Electricity prices are at all-time record highs. And when it comes to oil and gasoline, an extraordinarily small % of vehicles run on electricity; Tesla hasn't boosted GDP growth what-so-ever yet, their scale simply is not big enough. I see no evidence that an oil collar has been removed, nor that solar is meaningfully boosting GDP growth at this point.
beat 12y ago
I live just across the river from a recently-retired Ford plant in St Paul. The plant had its own dedicated hydroelectric dam on the Mississippi (before you're shocked, the Mississippi just isn't very big in Minneapolis as rivers go, and it was whitewater there before being developed). I keep waiting for someone to buy up that location to build a giant data center - Amazon, Google, someone like that. It's a beautiful spot with clean dedicated power, in the heart of a metro area full of enterprise operations talent.
deleted 12y ago
[deleted]
- kens 12y ago
  By inverter, do you mean DC to AC conversion? I don't understand why datacenters or cooling systems would use inverters. Or are you talking about switching power supplies? https://en.wikipedia.org/wiki/Power_inverter https://en.wikipedia.org/wiki/Power_inverter
  - marcosdumay 12y ago
    Whatever the GP meant, it's really a shame that we get 60Hz AC, convert it to DC, just to convert it back to 60Hz AC, and send it down the line where it'll get converted to DC again, just to be converted to 120kHz AC and to DC another time. Just pushing that intermediate 60Hz AC into 120kHz would already bring a huge benefit. But we should be looking at ways to bypass that first conversion entirely, and increasing the size / reducing the number of those second to last AC to DC converters at a minimum.
listic 12y ago
FWIW, the site is blocked in Russia, though the most regular proxy works around it just fine, yet. I wonder what have the author written to trigger the censorship filter. The halter of censorship is pressing ever stronger. Пора валить :(

63 points by luu 12y ago

Actually that is still a pretty good location for steel mills, as well as other industries, like Bitcoin mining farms (perhaps those fall under datacenters...).

imglorp 12y ago

Being near a river wouldn't hurt for heat dissipation either.

_dark_matter_ 12y ago
Not great for the wildlife, however.
wyldfire 12y ago
Using a waterway as a heat sink for a server farm may not sound as bad as dumping waste, but it's bound to have an impact on the wildlife there.
- TheGRS 12y ago
  I think the heat alone would disrupt a river ecosystem.
thrownaway2424 12y ago
Modern datacenters are not cooled that way, and some of them don't even have substantial makeup water inputs.

bsder 12y ago

Unfortunately, the baseline assumption isn't correct. Actually, we built steel mills near cheap transportation, not cheap power. Why? Because the main power source for primary steelmaking is coal (coke, actually), not electricity. And you need cheap transport for the power source (coal), raw materials (iron ore and such), and the finished steel. And cheap transportation, water, helps with getting rid of heat. Electricity is mostly used in secondary steelmaking where you are taking already existing steel, remelting it, and adjusting its properties. Now, aluminum production is very much tied to cheap electricity. I seem to remember that Iceland does a tremendous amount of aluminum refining because of the cheap geothermal power.

twotwotwo 12y ago
> Unfortunately, the baseline assumption isn't correct. To quibble with that quibble, though the steel mills thing is in the title, I wouldn't call it a "baseline assumption" of the longer discussion in the post: the post talks about how power affects datacenter costs, with one sentence at the end mentioning steel mills. He links the "steel mills" phrase to a page on a Microsoft datacenter in Quincy, WA. I don't know much about the history of that area, but I did read that another Microsoft datacenter was built in The Dalles, OR, which had very cheap power and historically was home to aluminum manufacturers that needed a lot of power for the electrolytic process involved: http://www.nytimes.com/2006/06/14/technology/14search.html?ei=&_r=0 http://www.nytimes.com/2006/06/14/technology/14search.html?e... mentions the datacenter and that aluminum smelters were once in the area http://en.wikipedia.org/wiki/Aluminium_smelting http://en.wikipedia.org/wiki/Aluminium_smelting talks more about the process and how it takes "prodigious amounts of electricity"
- butchlugrod 12y ago
  Quincy, like The Dalles, is located near the Columbia River, and is an hour drive to the Grand Coulee, Chief Joseph, Wanapum, Rock Island, and Rocky Reach hydroelectric dams. Electricity in central Washington is very inexpensive. Since that article was written, Dell, Yahoo, Vantage, and Sabey have all constructed data centers in the Quincy area.
  - AllexS 12y ago
    Also note that a very large reason for constructing Rocky Reach Dam on the Columbia was to provide power for an Alcoa Aluminum plant that is still heavily dependent on cheap energy prices. Chelan, Douglas and Grant counties all have nearly the cheapest power rates the the country.
tracker1 12y ago
Beyond that, I'd say that low-latency abundant bandwidth availability is probably harder to find and bring into a site than power is these days. Not to mention effective cooling. Then again, there's some very large banking institutions with data centers here in the Phoenix area, so what do I know.

twotwotwo 12y ago

++luu for an interesting read. One thought on hardware acceleration: it may be kind of happening, piece by piece. AES-NI, carry-less multiply, and CRC32 instructions are kind of like smallish (in gate area) accelerators of operations that go much easier in hardware. AMD talked about a compression accelerator for their (now-delayed) A1100 ARM server chip. These are marketed as performance features, not as power-saving, but if you can do something in fewer clock cycles without a ton of area, you are often saving on power as well. It's also possible that some savings come through finding places where slower chips will do the job because RAM and/or disk and/or bandwidth is the limiter (storage, caching, maybe some proxies and scale-out DBs now running on faster chips). Intel has server Atoms (Avoton, Rangeley) and there are experiments with ARM servers (Baidu cloud storage tried them out: https://gigaom.com/2013/02/20/first-arm-based-servers-in-production-support-baidus-cloud-storage/ https://gigaom.com/2013/02/20/first-arm-based-servers-in-pro...). There's lots of work on ARMs and Atoms on the consumer side and some of the benefits may spill over to low-power servers. Don't know how widespread this will be, though; I've been rooting for the concept of servers on low-power chips for years and haven't seen much yet. You also pay for higher power dissipation with lower density. If you're Amazon or some huge provider, you can be talking about a lot of space in absolute terms. It's interesting to eyeball a latest-gen Facebook Open Compute server with high-wattage CPUs in it, and how much of its height seems to be due to those heatsinks and fans: http://images.anandtech.com/doci/9138/leopard-mob.jpg http://images.anandtech.com/doci/9138/leopard-mob.jpg SeaMicro sadly didn't succeed, but it's interesting they thought people wanted density enough to make it a key part of their marketing, and AMD thought enough of the idea to buy them: http://www.brightsideofnews.com/Data/2012_9_11/AMD-Expands-SeaMicro-Big-Data-Gets-a-High-Performance-Home/AMD_SeaMicro7_689.jpg http://www.brightsideofnews.com/Data/2012_9_11/AMD-Expands-S... Of course, companies using and selling blade servers and the popularity of <1U configs (e.g., SuperMicro's twin servers: http://www.supermicro.com/products/nfo/1UTwin.cfm http://www.supermicro.com/products/nfo/1UTwin.cfm, Microsoft's blade-y variation on the idea: http://www.anandtech.com/show/9138/open-compute-hardware-tried-and-tested/9 http://www.anandtech.com/show/9138/open-compute-hardware-tri..., even that Facebook Open Compute layout which fits 3 boxes in 2U) demonstrate some interest in density.

willholloway 12y ago

Solar panel prices have dropped to a point where we are about to enter an era of energy cost deflation. This is an incredible human achievement, and the climate and economic benefits will be enormous. The US was muddling its way out the 2008 crash repercussions, with incremental improvement in the economy. It wasn't until the oil-choke collar on the economy was removed that we saw a return to the low unemployment levels we are seeing today. Energy costs are a tax on everything we do, and we are in for a global energy tax cut. The American Southwest has tremendous solar resources, and could be the cheapest place to run data centers in the states if we could develop: 1. Lower cost electricity storage, compressed air, new batteries, or superconducting magnetic energy storage could do it. 2. Develop economical solar thermal air conditioning systems. If you can generate heat, you can generate cold. http://en.wikipedia.org/wiki/Solar_air_conditioning http://en.wikipedia.org/wiki/Solar_air_conditioning

outworlder 12y ago
Where can I source panels that cheap, if I'm not in the US. Genuine question, I'd really want to know.
- toomuchtodo 12y ago
  China.
- grecy 12y ago
  My parents in Australia just paid $10k AUD for 5KW fully installed including the inverter and "smart meter". (the government later reimbursed them $5k of that, though that's not really relevant)
adventured 12y ago
Your economic references are incorrect. 1) Unemployment levels are not low, yet. U6 is about 11%. It was 8% in 2007, and 7% in 2001. The labor force participation rate is at a 40 year low. Full-time employment has seen zero net gain in over a decade, despite a population increase of 30 million people. 2) US GDP growth has seen zero improvement. The GDP growth rate the last five years is lower than it has been in all of US history outside of recessions (the average rate of growth from the 1940s to 2007 was about 3.5% +/-). There has been no acceleration. Growth rates - 2010: 2.5%; 2011: 1.6%; 2012: 2.3%; 2013: 2.2%; 2014: 2.4% These growth rates required 0% interest rates, $8 trillion in new public debt, and trillions in Fed stimulus. This latest quarter required a -0.1% inflation adjuster (only the second time that has happened in 65 years...) to keep the economy from printing another negative GDP growth quarter. The oil and fossil fuels collar has not been removed from the US economy. Electricity prices are at all-time record highs. And when it comes to oil and gasoline, an extraordinarily small % of vehicles run on electricity; Tesla hasn't boosted GDP growth what-so-ever yet, their scale simply is not big enough. I see no evidence that an oil collar has been removed, nor that solar is meaningfully boosting GDP growth at this point.

beat 12y ago

I live just across the river from a recently-retired Ford plant in St Paul. The plant had its own dedicated hydroelectric dam on the Mississippi (before you're shocked, the Mississippi just isn't very big in Minneapolis as rivers go, and it was whitewater there before being developed). I keep waiting for someone to buy up that location to build a giant data center - Amazon, Google, someone like that. It's a beautiful spot with clean dedicated power, in the heart of a metro area full of enterprise operations talent.

deleted 12y ago

[deleted]

kens 12y ago
By inverter, do you mean DC to AC conversion? I don't understand why datacenters or cooling systems would use inverters. Or are you talking about switching power supplies? https://en.wikipedia.org/wiki/Power_inverter https://en.wikipedia.org/wiki/Power_inverter
- marcosdumay 12y ago
  Whatever the GP meant, it's really a shame that we get 60Hz AC, convert it to DC, just to convert it back to 60Hz AC, and send it down the line where it'll get converted to DC again, just to be converted to 120kHz AC and to DC another time. Just pushing that intermediate 60Hz AC into 120kHz would already bring a huge benefit. But we should be looking at ways to bypass that first conversion entirely, and increasing the size / reducing the number of those second to last AC to DC converters at a minimum.

listic 12y ago

FWIW, the site is blocked in Russia, though the most regular proxy works around it just fine, yet. I wonder what have the author written to trigger the censorship filter. The halter of censorship is pressing ever stronger. Пора валить :(