5 ms·

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need

by gr2020 10y ago

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the rack, model what will happen if you lose one of the power lines, and all of the load switches to the other. You don't want to blow the circuit breaker on the other line, and lose the entire rack. 3. Thinking about steady state power is fine, but remember you may need to power the entire rack at full power in the worst case. Possibly from only one power feed. Make sure you have excess capacity for this. The first time I made a significant deployment of physical servers into a colo facility, power and cooling was quite literally the last thing I thought about. I'm guessing this is true for you too, based on the number of words you wrote about power. After several years of experience, power/cooling was almost the only thing I thought about.

jasoncchild 10y ago
These are all points a competent engineer would raise :)
- mverwijs 10y ago
  I've found that being considered a 'competent engineer' merely means 'never too arrogant to learn'. I didn't know half of the stuff in grandparents' post.
  - jasoncchild 10y ago
    I was attempting to confirm that these are all points are important. Being too arrogant to learn and being in incompetent are two entirely different things.
    - mverwijs 10y ago
      Sorry, I misunderstood. To me, your response came across as a "well, duh!".
      - jasoncchild 10y ago
        No problem; as I read it back to myself I can see how it came across that way.
spydum 10y ago
fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.
- sytse 10y ago
  Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.
  - DanielDent 10y ago
    You can certainly do some math/estimates based on looking at individual component specifications, but I like using a power meter (built in to some PDUs - which is a feature worth having, or you can buy one, they are quite inexpensive). A system at idle vs full CPU vs full cpu + all disk will produce very different measurements. Also keep in mind 80% derating - many electrical codes will state that an X amp circuit should only be used at 0.8X on a continuous basis (and the circuit breakers will be sized accordingly).
  - KenoFischer 10y ago
    I recently had to do this. The server I was putting up was rated for 3kW. To determine the expected load, I put it under a dummy load that I reasonably considered the maximum for what I would expect on the server (this was a dev machine, so I picked compiling the linux kernel as a benchmark). I ran that until the power stabilized (SuperMicro servers can measure power consumption in hardware and expose this via IPMI - very handy), because power consumption may keep creeping up for a few minutes as the fans adjust to the new operating temperature. I then repeated the same exercise with a CPU torture test (Prime95), just to see what the maximum I could possible get out of the machine was. The numbers turned out to be about 1.8kW for the linux kernel benchmark, and 2.3kW for the torture test. What I ended up doing was to provision for 2kW and use the BIOS' power limiting feature to enforce this. That would kill the machine if it exceeded its designed load, but that's usually better than tripping the breaker and killing the entire circuit. You may also want to talk to your data center provider about overrages. Some of the quotes I got wouldn't kill your circuit when you went over, but would just charge you (a crazy amount, but better than losing all your servers). Hope that helps. Your deployment is larger than ours, so there may be other techniques, but that's what we did.
gr2020 10y ago
I just remembered a blog post I wrote a while back about exactly these points: http://www.rassoc.com/gregr/weblog/2011/04/13/choosing-a-colocation-vendor-power-and-cooling-part-3/ http://www.rassoc.com/gregr/weblog/2011/04/13/choosing-a-col...
- sytse 10y ago
  Thanks!
davidbrent 10y ago
Think much harder about power and cooling. A few points: I've only ever built desktop machines, and this top comment drew a surprising parallel to most help me with my desktop build type posts. Granted, I'm sure as you dig deeper, the reasoning may be much different, but myself being ignorant about a proper server build, it was somehow reassuring to see power and cooling at the top!
late2part 10y ago
Nowadays once you get past 10 racks or 50kw, you generally only pay for power - the space/cooling/etc is "free" as your limiting rate is power and the vendor's ability to move thermal. You'll likely want a chimney cabinet like the Chatsworth GlobalFrame [1]. [1] - http://www.chatsworth.com/products/cabinet-and-enclosure-systems/globalframe-cabinets/ http://www.chatsworth.com/products/cabinet-and-enclosure-sys...
- walrus01 10y ago
  This does depend a lot on geographical location. You're not going to get free racks in a major carrier exchange point in Manhattan, or downtown Seattle, or close to the urban core of San Francisco. There will definitely be a significant quantity discount, as you increase in numbers, and a lot of your cost will be power and not racks. In somewhere that is very close (OSI layer 1) topologically to a major traffic interexchange point, you will definitely be paying somehow for the monthly cost of the square footage occupied. For example a colo in 60 Hudson or 25 Broadway in NYC, or in one Wilshire in LA. Large-scale colocation pricing that is based on power only will be found in locations that are not also major traffic inter exchange points. For example quincy WA or the many data centers in suburban new jersey.
  - late2part 10y ago
    I think your point is right for 111 8th or Equinix/SV1/Great Oaks. But for other sites with high value peering, including EQIX in Ashburn, Coresite, your limiting rate is likely to be the power, not the space or power. I.e. they'll "give" you 1 cabinet for every 15kw you buy. So my assertion assumes you're doing large number of dense cabinets. If one is doing a large number of dense cabinets, they almost certainly should not do it at a high value peering point, and should backhaul it. You should be able to get diverse metro dark fiber for <$5k/mo if not substantially less. Put a single cabinet (or pair for redundancy) of $2k/mo cabinets in Equinix or Fisher and off you go.
walrus01 10y ago
Something you should do to understand the actual power consumption of a server (ac power, in watts , at its input): Build one node with the hardware configuration you intend to use. Same CPU, ram, storage. Put it on a watt meter accurate to 1W. Install Debian amd64 on it in a basic config and run 256 threads of the 'cpuburn' package while simultaneously running iozone disk bench and memory benchmarks. This will give the figure of the absolute maximum load of the server, in watts, when it is running at 100% load on all cores, memory IO, and disk IO. Watts = heat, since all electricity consumed in a data center is either being used to do physical work like spinning a fan, or is going to end up in the air as heat. Laws of physics. As whatever data center you're using will be responsible for cooling, this is not exactly your problem, but you should be aware of it if you're going to try to do something like 12 kilowatts density per rack. Then multiply wattage of your prototype unit by number of servers. This will tell you how many fully loaded systems will fit in one 208v 30a circuit on the AC side. Also, use the system bios option for power recovery to stagger bootup times by 10 seconds each per server, so that in the event of a total power loss and an entire rack of servers does not attempt to power up simultaneously.
- btgeekboy 10y ago
  This is good advice. I'd consider not having them auto power on, however. This allows them to bring things up in a controlled manner. That brings me to my next point: GitLab should also being mindful on what services are stored on which hardware - performing heroics to work around circular dependencies is the last thing you want to be doing when recovering from a power outage.
  - walrus01 10y ago
    Yes, this is a choice that depends on the facility, and what sort of recovery plan you have for total power failure. In many cases you would want to have everything remain off, and have a remote hands person power up certain Network equipment, and key service, before everything else. On the other hand, you may want to design everything to recover itself, without any button pushing from humans. Depends a lot of server/HA/software architecture.

by gr2020 10y ago

jasoncchild 10y ago

These are all points a competent engineer would raise :)

mverwijs 10y ago
I've found that being considered a 'competent engineer' merely means 'never too arrogant to learn'. I didn't know half of the stuff in grandparents' post.
- jasoncchild 10y ago
  I was attempting to confirm that these are all points are important. Being too arrogant to learn and being in incompetent are two entirely different things.
  - mverwijs 10y ago
    Sorry, I misunderstood. To me, your response came across as a "well, duh!".
    - jasoncchild 10y ago
      No problem; as I read it back to myself I can see how it came across that way.

spydum 10y ago

fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.

sytse 10y ago
Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.
- DanielDent 10y ago
  You can certainly do some math/estimates based on looking at individual component specifications, but I like using a power meter (built in to some PDUs - which is a feature worth having, or you can buy one, they are quite inexpensive). A system at idle vs full CPU vs full cpu + all disk will produce very different measurements. Also keep in mind 80% derating - many electrical codes will state that an X amp circuit should only be used at 0.8X on a continuous basis (and the circuit breakers will be sized accordingly).
- KenoFischer 10y ago
  I recently had to do this. The server I was putting up was rated for 3kW. To determine the expected load, I put it under a dummy load that I reasonably considered the maximum for what I would expect on the server (this was a dev machine, so I picked compiling the linux kernel as a benchmark). I ran that until the power stabilized (SuperMicro servers can measure power consumption in hardware and expose this via IPMI - very handy), because power consumption may keep creeping up for a few minutes as the fans adjust to the new operating temperature. I then repeated the same exercise with a CPU torture test (Prime95), just to see what the maximum I could possible get out of the machine was. The numbers turned out to be about 1.8kW for the linux kernel benchmark, and 2.3kW for the torture test. What I ended up doing was to provision for 2kW and use the BIOS' power limiting feature to enforce this. That would kill the machine if it exceeded its designed load, but that's usually better than tripping the breaker and killing the entire circuit. You may also want to talk to your data center provider about overrages. Some of the quotes I got wouldn't kill your circuit when you went over, but would just charge you (a crazy amount, but better than losing all your servers). Hope that helps. Your deployment is larger than ours, so there may be other techniques, but that's what we did.

gr2020 10y ago

I just remembered a blog post I wrote a while back about exactly these points: http://www.rassoc.com/gregr/weblog/2011/04/13/choosing-a-colocation-vendor-power-and-cooling-part-3/ http://www.rassoc.com/gregr/weblog/2011/04/13/choosing-a-col...

sytse 10y ago
Thanks!

davidbrent 10y ago

Think much harder about power and cooling. A few points: I've only ever built desktop machines, and this top comment drew a surprising parallel to most help me with my desktop build type posts. Granted, I'm sure as you dig deeper, the reasoning may be much different, but myself being ignorant about a proper server build, it was somehow reassuring to see power and cooling at the top!

late2part 10y ago

Nowadays once you get past 10 racks or 50kw, you generally only pay for power - the space/cooling/etc is "free" as your limiting rate is power and the vendor's ability to move thermal. You'll likely want a chimney cabinet like the Chatsworth GlobalFrame [1]. [1] - http://www.chatsworth.com/products/cabinet-and-enclosure-systems/globalframe-cabinets/ http://www.chatsworth.com/products/cabinet-and-enclosure-sys...

walrus01 10y ago
This does depend a lot on geographical location. You're not going to get free racks in a major carrier exchange point in Manhattan, or downtown Seattle, or close to the urban core of San Francisco. There will definitely be a significant quantity discount, as you increase in numbers, and a lot of your cost will be power and not racks. In somewhere that is very close (OSI layer 1) topologically to a major traffic interexchange point, you will definitely be paying somehow for the monthly cost of the square footage occupied. For example a colo in 60 Hudson or 25 Broadway in NYC, or in one Wilshire in LA. Large-scale colocation pricing that is based on power only will be found in locations that are not also major traffic inter exchange points. For example quincy WA or the many data centers in suburban new jersey.
- late2part 10y ago
  I think your point is right for 111 8th or Equinix/SV1/Great Oaks. But for other sites with high value peering, including EQIX in Ashburn, Coresite, your limiting rate is likely to be the power, not the space or power. I.e. they'll "give" you 1 cabinet for every 15kw you buy. So my assertion assumes you're doing large number of dense cabinets. If one is doing a large number of dense cabinets, they almost certainly should not do it at a high value peering point, and should backhaul it. You should be able to get diverse metro dark fiber for <$5k/mo if not substantially less. Put a single cabinet (or pair for redundancy) of $2k/mo cabinets in Equinix or Fisher and off you go.

walrus01 10y ago

Something you should do to understand the actual power consumption of a server (ac power, in watts , at its input): Build one node with the hardware configuration you intend to use. Same CPU, ram, storage. Put it on a watt meter accurate to 1W. Install Debian amd64 on it in a basic config and run 256 threads of the 'cpuburn' package while simultaneously running iozone disk bench and memory benchmarks. This will give the figure of the absolute maximum load of the server, in watts, when it is running at 100% load on all cores, memory IO, and disk IO. Watts = heat, since all electricity consumed in a data center is either being used to do physical work like spinning a fan, or is going to end up in the air as heat. Laws of physics. As whatever data center you're using will be responsible for cooling, this is not exactly your problem, but you should be aware of it if you're going to try to do something like 12 kilowatts density per rack. Then multiply wattage of your prototype unit by number of servers. This will tell you how many fully loaded systems will fit in one 208v 30a circuit on the AC side. Also, use the system bios option for power recovery to stagger bootup times by 10 seconds each per server, so that in the event of a total power loss and an entire rack of servers does not attempt to power up simultaneously.

btgeekboy 10y ago
This is good advice. I'd consider not having them auto power on, however. This allows them to bring things up in a controlled manner. That brings me to my next point: GitLab should also being mindful on what services are stored on which hardware - performing heroics to work around circular dependencies is the last thing you want to be doing when recovering from a power outage.
- walrus01 10y ago
  Yes, this is a choice that depends on the facility, and what sort of recovery plan you have for total power failure. In many cases you would want to have everything remain off, and have a remote hands person power up certain Network equipment, and key service, before everything else. On the other hand, you may want to design everything to recover itself, without any button pushing from humans. Depends a lot of server/HA/software architecture.