5 ms·

None of this math is correct. You absolutely cannot get 128 cores for $500. Please link to a single valid-looking eBay offer at that amount (you may have seen

None of this math is correct. You absolutely cannot get 128 cores for $500. Please link to a single valid-looking eBay offer at that amount (you may have seen 128GB and thought it was 128 cores). I see 32 cores for $500, at best. Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo, not well over 10k. Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. Last, and most importantly, most people simply don't run servers 24/7. They run batch computations for five hours a week, or a day, and then spend a week or two crunching the numbers. There's some case to make that for certain institutions with very particular compute needs, running on-premise might make short-term financial sense (let's not even get started on labor costs). But it's really inaccurate to call cloud computing a "rip-off" for anything but the niche-est customers.

sigstoat 6y ago
> Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. not to mention the cooling, or the per square foot cost of renting the space for this hardware. anything close to what's being described here is going to need some sound deadening (be it distance or material) from humans.
- EB66 6y ago
  Frankly, I think many people here are overestimating the cost of electricity and cooling. My company leases several 42U cabinets with 208V 30A circuits in a modern SOC 2 Type 2 audited datacenter. We pay $600 per month per cabinet. Excludes IP transit costs. Granted this is in the Pacific Northwest where we have plentiful cheap hydro power.
  - rumanator 6y ago
    > We pay $600 per month per cabinet. Still a world of difference between AWS's 10k/month cost to access that many cores.
    - maccard 6y ago
      Comparing a Dell r740 (56 cores, 192GB ram, 1.6 kw psu), you need 11 of these for 640 cores (for a comparison with [0]). They're roughly 2500 each, so your up front hardware costs are 27,500. Assuming the $600 covers power, rack costs and cooling, your data center bill is 6600/month. For three years, the AWS bill is $315k, and the self hosted option is $265,100 (best case). I'm not up to speed with the pricing trends to say whether the gap will change over those 3 years based on data center changes or aws changes, but the difference is ~50k for three years, or roughly 3 months of an engineers salary over the cost of three months. [0] https://news.ycombinator.com/item?id=23514698 https://news.ycombinator.com/item?id=23514698
      - namibj 6y ago
        Did you consider that AWS counts threads, not cores?
        maccard 6y ago
        No, I did napkin math that compared an unverified number farther up the thread to the first server I found on Google. If you're considering buying hardware you should do your own numbers, I was merely pointing out that difference isn't 10k/month vs 27k up front
      - rumanator 6y ago
        > so your up front hardware costs are 27,500. That's about 3 months worth of AWS to access a similar amount of computational resources.
      - ies7 6y ago
        And if you don’t need them all the time , instead of 3 years RI you can us spot fleet or turn off half of the servers at non peak hour.
      - Dylan16807 6y ago
        It's $600 per cabinet, not $600 per server. If you go by the PSU, that's 3 cabinets. But that 1.6kW PSU is quite overspecced for multiple reasons, like efficiency and the fact that it derates to half on 110v power. 28 core xeons have a TDP in the 165-255 watt range, after all. So even 1kW per server is overkill and that's only 2 cabinets. $1200 per month, not $6600.
carlosf 6y ago
On a side note, you would expect people to be super excited to explore possibilities of new infrastructure models. Instead, most posts related to lambda, etc... always come with this sort of reply. For my use case (infrequent, easily separable jobs) FaaS stuff is a godsend. In the last year I've been doing things that would be orders of magnitude harder and more expensive thanks to it.
nullc 6y ago
> I see 32 cores for $500, at best. Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo, You can beat that cores per dollar handedly that if you're willing to use sufficiently odd hardware. E.g. my Intel cluster here is mostly I have systems with 4x E7-8880v4 (22c@2.2GHz). I paid $150/chip + $450 host (thanks ebay) excluding ram. (I have 8 such hosts now, so 704 cores) So essentially one months cost for your AWS price w/ a 3year commit pays for an equivalent core count in systems for me... Throw in another month for the ram. Round up to another another to count for the (quite) non-trivial power usage. You do not have to be anywhere near 100% utilization for AWS to be an extremely poor value large computing jobs compared to thrifty spending on the surplus market.
- what-the-grump 6y ago
  how? >I paid $150/chip + $450 host (thanks ebay) excluding ram. E7-8880v4 are $600 used each. Quad core mobos are not cheap, ranked ram is not cheap. Even if you pulled these from a vendor going out of business the bit coin mining crews would suck these up in heartbeat. Where are you getting your prices?
  - nullc 6y ago
    Ebay. Hunt harder (and best offer harder). :) E7 CPUs sell absurdly cheap at times because it can be a bit hard to find compatible systems. (similarly, quad socket systems with slow or no cpus go cheap from people that don't know much about them) > ranked ram is not cheap. Yes, ram for quad socket hosts it adds up, since you need at least 2-8 dimms per socket for full memory bandwidth (which was why I explicitly pointed out that I left it out). On the plus side, those systems tend to have a lot of dimm sockets so you can use lower capacity dimms. > bit coin mining crews nah, not useful for that.

sigstoat 6y ago

> Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. not to mention the cooling, or the per square foot cost of renting the space for this hardware. anything close to what's being described here is going to need some sound deadening (be it distance or material) from humans.

EB66 6y ago
Frankly, I think many people here are overestimating the cost of electricity and cooling. My company leases several 42U cabinets with 208V 30A circuits in a modern SOC 2 Type 2 audited datacenter. We pay $600 per month per cabinet. Excludes IP transit costs. Granted this is in the Pacific Northwest where we have plentiful cheap hydro power.
- rumanator 6y ago
  > We pay $600 per month per cabinet. Still a world of difference between AWS's 10k/month cost to access that many cores.
  - maccard 6y ago
    Comparing a Dell r740 (56 cores, 192GB ram, 1.6 kw psu), you need 11 of these for 640 cores (for a comparison with [0]). They're roughly 2500 each, so your up front hardware costs are 27,500. Assuming the $600 covers power, rack costs and cooling, your data center bill is 6600/month. For three years, the AWS bill is $315k, and the self hosted option is $265,100 (best case). I'm not up to speed with the pricing trends to say whether the gap will change over those 3 years based on data center changes or aws changes, but the difference is ~50k for three years, or roughly 3 months of an engineers salary over the cost of three months. [0] https://news.ycombinator.com/item?id=23514698 https://news.ycombinator.com/item?id=23514698
    - namibj 6y ago
      Did you consider that AWS counts threads, not cores?
      - maccard 6y ago
        No, I did napkin math that compared an unverified number farther up the thread to the first server I found on Google. If you're considering buying hardware you should do your own numbers, I was merely pointing out that difference isn't 10k/month vs 27k up front
    - rumanator 6y ago
      > so your up front hardware costs are 27,500. That's about 3 months worth of AWS to access a similar amount of computational resources.
    - ies7 6y ago
      And if you don’t need them all the time , instead of 3 years RI you can us spot fleet or turn off half of the servers at non peak hour.
    - Dylan16807 6y ago
      It's $600 per cabinet, not $600 per server. If you go by the PSU, that's 3 cabinets. But that 1.6kW PSU is quite overspecced for multiple reasons, like efficiency and the fact that it derates to half on 110v power. 28 core xeons have a TDP in the 165-255 watt range, after all. So even 1kW per server is overkill and that's only 2 cabinets. $1200 per month, not $6600.

carlosf 6y ago

On a side note, you would expect people to be super excited to explore possibilities of new infrastructure models. Instead, most posts related to lambda, etc... always come with this sort of reply. For my use case (infrequent, easily separable jobs) FaaS stuff is a godsend. In the last year I've been doing things that would be orders of magnitude harder and more expensive thanks to it.

nullc 6y ago

> I see 32 cores for $500, at best. Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo, You can beat that cores per dollar handedly that if you're willing to use sufficiently odd hardware. E.g. my Intel cluster here is mostly I have systems with 4x E7-8880v4 (22c@2.2GHz). I paid $150/chip + $450 host (thanks ebay) excluding ram. (I have 8 such hosts now, so 704 cores) So essentially one months cost for your AWS price w/ a 3year commit pays for an equivalent core count in systems for me... Throw in another month for the ram. Round up to another another to count for the (quite) non-trivial power usage. You do not have to be anywhere near 100% utilization for AWS to be an extremely poor value large computing jobs compared to thrifty spending on the surplus market.

what-the-grump 6y ago
how? >I paid $150/chip + $450 host (thanks ebay) excluding ram. E7-8880v4 are $600 used each. Quad core mobos are not cheap, ranked ram is not cheap. Even if you pulled these from a vendor going out of business the bit coin mining crews would suck these up in heartbeat. Where are you getting your prices?
- nullc 6y ago
  Ebay. Hunt harder (and best offer harder). :) E7 CPUs sell absurdly cheap at times because it can be a bit hard to find compatible systems. (similarly, quad socket systems with slow or no cpus go cheap from people that don't know much about them) > ranked ram is not cheap. Yes, ram for quad socket hosts it adds up, since you need at least 2-8 dimms per socket for full memory bandwidth (which was why I explicitly pointed out that I left it out). On the plus side, those systems tend to have a lot of dimm sockets so you can use lower capacity dimms. > bit coin mining crews nah, not useful for that.