5 ms·

I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage. AWS does not have a 100% similar VM, but yo

by pid-1 3y ago

I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage. AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. However, storage costs alone would be astronomic. Like > 100,000 USD / month. I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense. OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but: > I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month I would not take the other side on that bet

ttegloma 3y ago
> AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month. For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example. Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.
- Shakahs 3y ago
  They're also using ZFS in Raid1+0, so 38.4TB of usable storage. $14,566/month on AWS io2 with max IOPS.
jaas 3y ago
This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR. - ED of ISRG / Let's Encrypt
alberth 3y ago
It’s closer to $80k, if you went via a VAR. I spec’d a current gen server with: - 2x more cores - 50% more RAM - 50% more storage For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage. It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old). https://www.siliconmechanics.com/system/rackform-a335.v9 https://www.siliconmechanics.com/system/rackform-a335.v9
dharmab 3y ago
I'd also bet money that this has higher disk I/O performance than the rough equivalent on Amazon.
10000truths 3y ago
$225K for this sounds bonkers: 24x 6.4TB Intel P4610 NVMe SSD = 24 x $310 = $7440 2x AMD EPYC 7542 = 2 x $1300 = $2600 2 TB DDR4 ECC RAM ~ $13700 (estimate from a couple of Google results) Those add up to something like $25K. Sure, there's also the price of the motherboard, chassis, maybe some other peripherals like external network cards, assembly + support + warranty etc. but that doesn't explain an 800% markup.
- icedchai 3y ago
  For a critical system, you should really have two, for HA purposes. Do they?
  - mcpherrinm 3y ago
    Yes. Let’s Encrypt has two locations, each of which has fully redundant hardware, so that’s a minimum of 4. We actually have a few more. (I work at Let’s Encrypt)
- sseagull 3y ago
  We recently purchased a similar server. One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website. Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free. Even on Newegg, SAS SSD of that size are $900-2000, so add warranty and service on top of that.
  - 10000truths 3y ago
    > One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website. Makes sense. > Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free. I was able to find these two enterprise-grade NVMe SSDs on Newegg: https://www.newegg.com/p/2U3-0005-000J8 ($474, 7.68 TB, 1 DWPD) https://www.newegg.com/p/2U3-000S-000Y8 ($360, 6.4 TB, 3 DWPD) Is there some kind of catch I'm missing?
CathalMullan 3y ago
> These are expensive servers, crossing into six digits, but not $200k. https://news.ycombinator.com/item?id=25865967 https://news.ycombinator.com/item?id=25865967
Spooky23 3y ago
Nobody is paying base price for a box like that. I’d probably bid them against HPE and pay ~90-100k. If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.
riku_iki 3y ago
> AWS does not have a 100% similar VM does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..

by pid-1 3y ago

ttegloma 3y ago

> AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad. Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month. For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example. Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.

Shakahs 3y ago
They're also using ZFS in Raid1+0, so 38.4TB of usable storage. $14,566/month on AWS io2 with max IOPS.

jaas 3y ago

This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR. - ED of ISRG / Let's Encrypt

alberth 3y ago

It’s closer to $80k, if you went via a VAR. I spec’d a current gen server with: - 2x more cores - 50% more RAM - 50% more storage For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage. It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old). https://www.siliconmechanics.com/system/rackform-a335.v9 https://www.siliconmechanics.com/system/rackform-a335.v9

dharmab 3y ago

I'd also bet money that this has higher disk I/O performance than the rough equivalent on Amazon.

10000truths 3y ago

$225K for this sounds bonkers: 24x 6.4TB Intel P4610 NVMe SSD = 24 x $310 = $7440 2x AMD EPYC 7542 = 2 x $1300 = $2600 2 TB DDR4 ECC RAM ~ $13700 (estimate from a couple of Google results) Those add up to something like $25K. Sure, there's also the price of the motherboard, chassis, maybe some other peripherals like external network cards, assembly + support + warranty etc. but that doesn't explain an 800% markup.

icedchai 3y ago
For a critical system, you should really have two, for HA purposes. Do they?
- mcpherrinm 3y ago
  Yes. Let’s Encrypt has two locations, each of which has fully redundant hardware, so that’s a minimum of 4. We actually have a few more. (I work at Let’s Encrypt)
sseagull 3y ago
We recently purchased a similar server. One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website. Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free. Even on Newegg, SAS SSD of that size are $900-2000, so add warranty and service on top of that.
- 10000truths 3y ago
  > One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website. Makes sense. > Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free. I was able to find these two enterprise-grade NVMe SSDs on Newegg: https://www.newegg.com/p/2U3-0005-000J8 ($474, 7.68 TB, 1 DWPD) https://www.newegg.com/p/2U3-000S-000Y8 ($360, 6.4 TB, 3 DWPD) Is there some kind of catch I'm missing?

CathalMullan 3y ago

> These are expensive servers, crossing into six digits, but not $200k. https://news.ycombinator.com/item?id=25865967 https://news.ycombinator.com/item?id=25865967

Spooky23 3y ago

Nobody is paying base price for a box like that. I’d probably bid them against HPE and pay ~90-100k. If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.

riku_iki 3y ago

> AWS does not have a 100% similar VM does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..