8 ms·

Frankly, I think many people here are overestimating the cost of electricity and cooling. My company leases several 42U cabinets with 208V 30A circuits in a mod

by EB66 6y ago

Frankly, I think many people here are overestimating the cost of electricity and cooling. My company leases several 42U cabinets with 208V 30A circuits in a modern SOC 2 Type 2 audited datacenter. We pay $600 per month per cabinet. Excludes IP transit costs. Granted this is in the Pacific Northwest where we have plentiful cheap hydro power.

rumanator 6y ago
> We pay $600 per month per cabinet. Still a world of difference between AWS's 10k/month cost to access that many cores.
- maccard 6y ago
  Comparing a Dell r740 (56 cores, 192GB ram, 1.6 kw psu), you need 11 of these for 640 cores (for a comparison with [0]). They're roughly 2500 each, so your up front hardware costs are 27,500. Assuming the $600 covers power, rack costs and cooling, your data center bill is 6600/month. For three years, the AWS bill is $315k, and the self hosted option is $265,100 (best case). I'm not up to speed with the pricing trends to say whether the gap will change over those 3 years based on data center changes or aws changes, but the difference is ~50k for three years, or roughly 3 months of an engineers salary over the cost of three months. [0] https://news.ycombinator.com/item?id=23514698 https://news.ycombinator.com/item?id=23514698
  - namibj 6y ago
    Did you consider that AWS counts threads, not cores?
    - maccard 6y ago
      No, I did napkin math that compared an unverified number farther up the thread to the first server I found on Google. If you're considering buying hardware you should do your own numbers, I was merely pointing out that difference isn't 10k/month vs 27k up front
  - rumanator 6y ago
    > so your up front hardware costs are 27,500. That's about 3 months worth of AWS to access a similar amount of computational resources.
  - ies7 6y ago
    And if you don’t need them all the time , instead of 3 years RI you can us spot fleet or turn off half of the servers at non peak hour.
  - Dylan16807 6y ago
    It's $600 per cabinet, not $600 per server. If you go by the PSU, that's 3 cabinets. But that 1.6kW PSU is quite overspecced for multiple reasons, like efficiency and the fact that it derates to half on 110v power. 28 core xeons have a TDP in the 165-255 watt range, after all. So even 1kW per server is overkill and that's only 2 cabinets. $1200 per month, not $6600.
    - maccard 6y ago
      With your numbers, that puts the total running cost at 71k, or a saving of $245k over 3 years, which is pretty much the salary of one person to look after that system for the lifetime. I'm not arguing that AWS is good value, im arguing that the upfront cost of purchasing the hardware isn't comparable to reserving EC2 instances up front.
      - Dylan16807 6y ago
        You still need someone to look after your systems if you're using AWS. That's not something special to buying your own server. And management of the hardware itself doesn't take an extra employee. It probably averages to less than one workday per month.
      - rumanator 6y ago
        > With your numbers, that puts the total running cost at 71k, or a saving of $245k over 3 years To compare apples to apples, that hypothetical pricetag buys you the system's max capacity for the entire period. In AWS, the invoice from using any of their serverless offerings grows wildly even when utilization stays way below the resources of that hardware bundle. For instance, let's keep in mind that AWS charges: * API Gateway per million of requests, * Lambdas per timememory used by a single call, including the time it wasted with hot and cold starts, SQS per queued message, * WAF per request through the firewall, * Etc.
      - vidarh 6y ago
        > which is pretty much the salary of one person to look after that system for the lifetime. My average time spend in data centers when handling racks in two different location was around 3-4 days a year. Buying on-call, out-of-hours support is going to cost you a tiny fraction of a persons salary per year, either on retainer and/or based on hefty day rates. Hardware failures simply do not occur often enough for a rack or two worth of hardware to typically require enough maintenance to make it that expensive, and remote hands (people who act based on a ticket) are available in pretty much any data centre at low costs. If you get servers with IPMI etc. you typically pop in to wire up or retire equipment and dealing with the occasional failure. Source: Have provided those kind of services for years.

by EB66 6y ago

rumanator 6y ago

> We pay $600 per month per cabinet. Still a world of difference between AWS's 10k/month cost to access that many cores.

maccard 6y ago
Comparing a Dell r740 (56 cores, 192GB ram, 1.6 kw psu), you need 11 of these for 640 cores (for a comparison with [0]). They're roughly 2500 each, so your up front hardware costs are 27,500. Assuming the $600 covers power, rack costs and cooling, your data center bill is 6600/month. For three years, the AWS bill is $315k, and the self hosted option is $265,100 (best case). I'm not up to speed with the pricing trends to say whether the gap will change over those 3 years based on data center changes or aws changes, but the difference is ~50k for three years, or roughly 3 months of an engineers salary over the cost of three months. [0] https://news.ycombinator.com/item?id=23514698 https://news.ycombinator.com/item?id=23514698
- namibj 6y ago
  Did you consider that AWS counts threads, not cores?
  - maccard 6y ago
    No, I did napkin math that compared an unverified number farther up the thread to the first server I found on Google. If you're considering buying hardware you should do your own numbers, I was merely pointing out that difference isn't 10k/month vs 27k up front
- rumanator 6y ago
  > so your up front hardware costs are 27,500. That's about 3 months worth of AWS to access a similar amount of computational resources.
- ies7 6y ago
  And if you don’t need them all the time , instead of 3 years RI you can us spot fleet or turn off half of the servers at non peak hour.
- Dylan16807 6y ago
  It's $600 per cabinet, not $600 per server. If you go by the PSU, that's 3 cabinets. But that 1.6kW PSU is quite overspecced for multiple reasons, like efficiency and the fact that it derates to half on 110v power. 28 core xeons have a TDP in the 165-255 watt range, after all. So even 1kW per server is overkill and that's only 2 cabinets. $1200 per month, not $6600.
  - maccard 6y ago
    With your numbers, that puts the total running cost at 71k, or a saving of $245k over 3 years, which is pretty much the salary of one person to look after that system for the lifetime. I'm not arguing that AWS is good value, im arguing that the upfront cost of purchasing the hardware isn't comparable to reserving EC2 instances up front.
    - Dylan16807 6y ago
      You still need someone to look after your systems if you're using AWS. That's not something special to buying your own server. And management of the hardware itself doesn't take an extra employee. It probably averages to less than one workday per month.
    - rumanator 6y ago
      > With your numbers, that puts the total running cost at 71k, or a saving of $245k over 3 years To compare apples to apples, that hypothetical pricetag buys you the system's max capacity for the entire period. In AWS, the invoice from using any of their serverless offerings grows wildly even when utilization stays way below the resources of that hardware bundle. For instance, let's keep in mind that AWS charges: * API Gateway per million of requests, * Lambdas per timememory used by a single call, including the time it wasted with hot and cold starts, SQS per queued message, * WAF per request through the firewall, * Etc.
    - vidarh 6y ago
      > which is pretty much the salary of one person to look after that system for the lifetime. My average time spend in data centers when handling racks in two different location was around 3-4 days a year. Buying on-call, out-of-hours support is going to cost you a tiny fraction of a persons salary per year, either on retainer and/or based on hefty day rates. Hardware failures simply do not occur often enough for a rack or two worth of hardware to typically require enough maintenance to make it that expensive, and remote hands (people who act based on a ticket) are available in pretty much any data centre at low costs. If you get servers with IPMI etc. you typically pop in to wire up or retire equipment and dealing with the occasional failure. Source: Have provided those kind of services for years.