6 ms·

A power-supply with 70-80% efficiency (at, say, around 50% of the power-rating) would be a very, very crappy one. On the other hand, a DC/DC with 98% would be a

by cnvogel 13y ago

A power-supply with 70-80% efficiency (at, say, around 50% of the power-rating) would be a very, very crappy one. On the other hand, a DC/DC with 98% would be a very good one. So your comparison is not really fair. See e.g. the book mentioned below. The same book quotes overall inefficiencies from power line to the CPU from as low as 48% for the crappy-PSU + UPS + converters to up to 92% for the Google AC/DC+integrated PSU design. E.g. "The Datacenter as a Computer An Introduction to the Design of Warehouse-Scale Machines, Second Edition, Luiz André, Barroso Jimmy, Clidaras Urs Hölzle" page 52 (PDF page 70) figure 4.2 quotes AC/DC supplies with 79% at worst, but the other two numbers quoted are 89% and 92%. That's more reasonable. http://www.morganclaypool.com/doi/pdf/10.2200/S00516ED2V01Y201306CAC024 http://www.morganclaypool.com/doi/pdf/10.2200/S00516ED2V01Y2...

Aloha 13y ago
I'll admit I cherry picked a little. But the numbers I used are largely borne out on the link you provided. Look at page 52 and 53. I think however larger rack/frame mounted DC/DC converters would likely be even more efficient than per chassis DC/DC converters, but would require almost totally custom hardware.
- lsc 13y ago
  >I think however larger rack/frame mounted DC/DC converters would likely be even more efficient than per chassis DC/DC converters, but would require almost totally custom hardware. Nope, you can buy off the shelf ATX PSUs that accept 48v DC as input, though it's kinda hard to find. It's easier to find, usually, on the higher-end. For switching gear, though? it's a standard option, as 48v DC is standard for telcos. In fact, most of the higher-end datacenters will sell you 48v DC if you ask for it. It's sort of like having an always-online UPS, as they can just leave the battery-banks in-line; no switching from mains power to UPS required. (and that switch is... quite often what fails when a data center loses power.) Whenever I take folks through my setup at coresite santa clara, they are always drawn to the 48vdc battery bank on the other side of the room. "Hey guys, that's not mine." "No, don't touch that giant copper bus bar." (That's the thing about low voltage; you need serious amounts of copper if you want to move very many watts at all.) It used to be a bigger deal than it is now, as the PSU in your server is dramatically better now than it was even just ten years ago. But yeah, from what I can tell? (and I spend what, 10x rent on data-center power? I'm not saying I'm an expert, but I certainly have an incentive to be right.) The negotiation differences between the cost per watt using different voltages is far greater than the efficiency differences of different voltages. Sales people ruin everything. At many places, 208v cost you twice as much per amp as 120v, so you are better off just buying twice as many 120v amps. (I mean, 208v is more efficient for you and for the datacenter, but it's not 10% more efficient, and you get more than 10% more watts that way.) I was at one place where an amp of 208v was only about 30% more expensive than an amp of 120v. Too bad that free ride ended. The problem is that most salespeople don't really understand any of this... they understand how to sell. And a 10% difference in power cost, well, that's less than the referral bonus they give if you have an agreement to feed them leads. (this is why you see so many small companies claiming to have datacenters across the country. They just have referral programs, and they get upwards of 10% of what the customer ends up paying the real data center, essentially in exchange for some slimy seo.) So really, the negotiation game matters a whole lot more than what is actually the most efficient, until you get to the point where you can build your own data-center.
  - Aloha 13y ago
    I'm aware of the neg 48 vdc stuff out there, I work in the telecom industry, and plan on building a 48v power plant whenever I buy a house :-P. That's not quite what I was talking about - Rather than need to distribute 48v you could do 240v or 384v (5 or 8 48v strings) and then do DC-DC in each frame to generate 12v/5v/3.3v power supplies, which supply the voltages directly to the hardware, if you needed 48v in frames, its easy to put a DC-DC converter in to get that too. I shudder to think at copper costs to distribute 48v.
    - lsc 13y ago
      huh. Interesting. How would the safety implications of running 240v DC differ from running 240v AC? (my understanding is that the amount of copper required would be the same.) From what I've seen, 48vdc is usually just two leads sticking out from the equipment in question.. you just strip the ends of the wires and screw 'em in. I don't think I'd be comfortable doing that with 240v. I guess that's just a simple matter of standardization.
      - Aloha 13y ago
        If you get bit (shocked) by DC, you let go, drop it instantly - AC will not allow you to drop it, you have to be pulled off. If anything its safer. Also, DC has lower loss over a given conductor. The problem historically with DC was changing from a high voltage for transmission and low voltages for loads, something changed by solid state devices.
        lsc 13y ago
        >If you get bit (shocked) by DC, you let go, drop it instantly - AC will not allow you to drop it, you have to be pulled off. If anything its safer. I know I was once shocked by 120v AC while working; I was reaching around a metal PDU with 15-15R plugs to plug something in. As I couldn't see, I was guiding the prongs in with my fingers, and I didn't get my fingers out of the way before it made contact. ouch. (let us just say that I was younger and... a lot less careful than a sysadmin ought to be.) Fortunately, the case of the PDU was grounded (I think it was a baytech RPC) and the case of the server above it was grounded, and everything was screwed in well, so the circut went through my fingers and arm and into a chassis. my impression was that my arm bounced between the PDU and the next server up for a while before I managed to jerk my shoulder back and pull my whole arm out of the rack. Apparently I'm not a very good conductor, or there was something up with the breaker, 'cause it didn't blow the circuit. I wasn't hurt, save for maybe some light bruising, and no downtime resulted, so it turned out okay, but I'm much more careful now (and my PDUs use C14 receptacles, making this problem impossible, at least without a knife.)

by cnvogel 13y ago

Aloha 13y ago

I'll admit I cherry picked a little. But the numbers I used are largely borne out on the link you provided. Look at page 52 and 53. I think however larger rack/frame mounted DC/DC converters would likely be even more efficient than per chassis DC/DC converters, but would require almost totally custom hardware.

lsc 13y ago
>I think however larger rack/frame mounted DC/DC converters would likely be even more efficient than per chassis DC/DC converters, but would require almost totally custom hardware. Nope, you can buy off the shelf ATX PSUs that accept 48v DC as input, though it's kinda hard to find. It's easier to find, usually, on the higher-end. For switching gear, though? it's a standard option, as 48v DC is standard for telcos. In fact, most of the higher-end datacenters will sell you 48v DC if you ask for it. It's sort of like having an always-online UPS, as they can just leave the battery-banks in-line; no switching from mains power to UPS required. (and that switch is... quite often what fails when a data center loses power.) Whenever I take folks through my setup at coresite santa clara, they are always drawn to the 48vdc battery bank on the other side of the room. "Hey guys, that's not mine." "No, don't touch that giant copper bus bar." (That's the thing about low voltage; you need serious amounts of copper if you want to move very many watts at all.) It used to be a bigger deal than it is now, as the PSU in your server is dramatically better now than it was even just ten years ago. But yeah, from what I can tell? (and I spend what, 10x rent on data-center power? I'm not saying I'm an expert, but I certainly have an incentive to be right.) The negotiation differences between the cost per watt using different voltages is far greater than the efficiency differences of different voltages. Sales people ruin everything. At many places, 208v cost you twice as much per amp as 120v, so you are better off just buying twice as many 120v amps. (I mean, 208v is more efficient for you and for the datacenter, but it's not 10% more efficient, and you get more than 10% more watts that way.) I was at one place where an amp of 208v was only about 30% more expensive than an amp of 120v. Too bad that free ride ended. The problem is that most salespeople don't really understand any of this... they understand how to sell. And a 10% difference in power cost, well, that's less than the referral bonus they give if you have an agreement to feed them leads. (this is why you see so many small companies claiming to have datacenters across the country. They just have referral programs, and they get upwards of 10% of what the customer ends up paying the real data center, essentially in exchange for some slimy seo.) So really, the negotiation game matters a whole lot more than what is actually the most efficient, until you get to the point where you can build your own data-center.
- Aloha 13y ago
  I'm aware of the neg 48 vdc stuff out there, I work in the telecom industry, and plan on building a 48v power plant whenever I buy a house :-P. That's not quite what I was talking about - Rather than need to distribute 48v you could do 240v or 384v (5 or 8 48v strings) and then do DC-DC in each frame to generate 12v/5v/3.3v power supplies, which supply the voltages directly to the hardware, if you needed 48v in frames, its easy to put a DC-DC converter in to get that too. I shudder to think at copper costs to distribute 48v.
  - lsc 13y ago
    huh. Interesting. How would the safety implications of running 240v DC differ from running 240v AC? (my understanding is that the amount of copper required would be the same.) From what I've seen, 48vdc is usually just two leads sticking out from the equipment in question.. you just strip the ends of the wires and screw 'em in. I don't think I'd be comfortable doing that with 240v. I guess that's just a simple matter of standardization.
    - Aloha 13y ago
      If you get bit (shocked) by DC, you let go, drop it instantly - AC will not allow you to drop it, you have to be pulled off. If anything its safer. Also, DC has lower loss over a given conductor. The problem historically with DC was changing from a high voltage for transmission and low voltages for loads, something changed by solid state devices.
      - lsc 13y ago
        >If you get bit (shocked) by DC, you let go, drop it instantly - AC will not allow you to drop it, you have to be pulled off. If anything its safer. I know I was once shocked by 120v AC while working; I was reaching around a metal PDU with 15-15R plugs to plug something in. As I couldn't see, I was guiding the prongs in with my fingers, and I didn't get my fingers out of the way before it made contact. ouch. (let us just say that I was younger and... a lot less careful than a sysadmin ought to be.) Fortunately, the case of the PDU was grounded (I think it was a baytech RPC) and the case of the server above it was grounded, and everything was screwed in well, so the circut went through my fingers and arm and into a chassis. my impression was that my arm bounced between the PDU and the next server up for a while before I managed to jerk my shoulder back and pull my whole arm out of the rack. Apparently I'm not a very good conductor, or there was something up with the breaker, 'cause it didn't blow the circuit. I wasn't hurt, save for maybe some light bruising, and no downtime resulted, so it turned out okay, but I'm much more careful now (and my PDUs use C14 receptacles, making this problem impossible, at least without a knife.)