4 ms·

$60K Data Infrastructure vs. $2M Cloud Equivalent

27 points by mihaivinaga 11mo ago

and1-1 11mo ago
so… they built a 4PB cluster, 14k cores, 50 GPUs, and it costs them less than my prod Kubernetes cluster on GCP. feels bad.
- Growther0506 11mo ago
  dont worry, their real cost was their sanity
  - automaticlemon 11mo ago
    [dead]
td17191719 11mo ago
and what about the team of 1000 engineers you need to do the work of AWS support?
alexandra494 11mo ago
This is maybe the first “ditch the cloud” post that isn’t totally ideological. Makes sense for their scale + workload. I wish more companies ran the numbers instead of just accepting AWS bills as fate.
idk123456 11mo ago
the part about “predictable heavy workloads” should be bolded in neon. 90% of startups THINK they have predictable workloads. they do not. these guys actually do.
- stefan420 11mo ago
  i would argue 90% of startups think they are going to need autoscaling, invest in that, and then never reach the scale up point, while having spent the investment money already. besides the point of the article, tho. even if at the end of the day the numbers don't perfectly add up, i don't believe they are WAY OFF. we are paying for cables and HDDs man. "ancient" tech.
DaveEliot 11mo ago
idk man, running your own hardware sounds awesome until you have to replace a dead RAID controller at 3am. article doesn’t really mention that part.
- mihaivinaga 11mo ago
  Well, we do have some servers that we have in the office(the gpus servers, those have been a pain in the... mostly because intel cpus suck). But other then these the entire hardware part is managed by hetzner (thank you hetzner for being awesome)
iancu 11mo ago
good story, but "works for us, so it should work for everyone". most startups can't touch this.
Growther0506 11mo ago
the irony is cloud providers would love to sell a customer like this giant blocks of reserved compute. wonder if they actually negotiated or just said “lol too expensive” and bailed.
radu26 11mo ago
not sure i buy the “$2M/mo on cloud” number. did they actually run it or just plug public on-demand pricing into a spreadsheet? nobody pays sticker price at that scale.
- mihaivinaga 11mo ago
  That's a fair point, the scale discounts certainly matter. That said: We did in fact run our prototype on AWS, and early on they flagged our account because our VMs were under heavy CPU load (we were told we'd need to cut below ~50 % or risk suspension). When you switch to dedicated hardware, you're no longer paying for "noisy neighbours" or hypervisor over-provisioning, so the effective per-unit cost often cleaner fully justifies the shift.
alextofan 11mo ago
Really appreciate how this isn't just another "cloud vs bare metal" holy war.The "own the core, rent the periphery" approach is exactly the kind of nuanced thinking we need more of in these discussions. That said,important caveat for anyone reading this: Veridion is processing petabytes of data with predictable workloads. If you're still figuring out product-market fit or your usage patterns are all over the place, premature infrastructure optimization will absolutely kill you.The cloud's "boring reliability" has real value when you need to focus on building the actual product. The 30x multiplier is real, but it only kicks in at serious scale with stable, known workloads. Most startups aren't there yet, and that's fine.
automaticlemon 11mo ago
[dead]
as3413 11mo ago
ngl the cloud evangelists aren’t gonna like this one. this is basically “we did the math and cloud wasn’t worth it
martis2018 11mo ago
not saying they’re wrong but $2m → $60k is like… a 97% cost drop. that is either cloud pricing insanity or some very optimistic bare-metal accounting.
- general1465 11mo ago
  If they are going from AWS, that would be the cloud pricing insanity.
cs0min 11mo ago
can’t tell if they’re geniuses or just suffering a massive case of “we built a data center so now everything looks like a hardware nail."
florint 11mo ago
This is a super interesting analysis, I wonder what scale do you need to reach the point where the ease of adoption of AWS/GCloud is not worth it anymore. The article compares compute in each scenario, which makes sense, but I would take into account the speed to live as a secondary metric. I'm pretty sure cloud is faster, at least initially

27 points by mihaivinaga 11mo ago

and1-1 11mo ago

so… they built a 4PB cluster, 14k cores, 50 GPUs, and it costs them less than my prod Kubernetes cluster on GCP. feels bad.

Growther0506 11mo ago
dont worry, their real cost was their sanity
- automaticlemon 11mo ago
  [dead]

td17191719 11mo ago

and what about the team of 1000 engineers you need to do the work of AWS support?

alexandra494 11mo ago

This is maybe the first “ditch the cloud” post that isn’t totally ideological. Makes sense for their scale + workload. I wish more companies ran the numbers instead of just accepting AWS bills as fate.

idk123456 11mo ago

the part about “predictable heavy workloads” should be bolded in neon. 90% of startups THINK they have predictable workloads. they do not. these guys actually do.

stefan420 11mo ago
i would argue 90% of startups think they are going to need autoscaling, invest in that, and then never reach the scale up point, while having spent the investment money already. besides the point of the article, tho. even if at the end of the day the numbers don't perfectly add up, i don't believe they are WAY OFF. we are paying for cables and HDDs man. "ancient" tech.

DaveEliot 11mo ago

idk man, running your own hardware sounds awesome until you have to replace a dead RAID controller at 3am. article doesn’t really mention that part.

mihaivinaga 11mo ago
Well, we do have some servers that we have in the office(the gpus servers, those have been a pain in the... mostly because intel cpus suck). But other then these the entire hardware part is managed by hetzner (thank you hetzner for being awesome)

iancu 11mo ago

good story, but "works for us, so it should work for everyone". most startups can't touch this.

Growther0506 11mo ago

the irony is cloud providers would love to sell a customer like this giant blocks of reserved compute. wonder if they actually negotiated or just said “lol too expensive” and bailed.

radu26 11mo ago

not sure i buy the “$2M/mo on cloud” number. did they actually run it or just plug public on-demand pricing into a spreadsheet? nobody pays sticker price at that scale.

mihaivinaga 11mo ago
That's a fair point, the scale discounts certainly matter. That said: We did in fact run our prototype on AWS, and early on they flagged our account because our VMs were under heavy CPU load (we were told we'd need to cut below ~50 % or risk suspension). When you switch to dedicated hardware, you're no longer paying for "noisy neighbours" or hypervisor over-provisioning, so the effective per-unit cost often cleaner fully justifies the shift.

alextofan 11mo ago

Really appreciate how this isn't just another "cloud vs bare metal" holy war.The "own the core, rent the periphery" approach is exactly the kind of nuanced thinking we need more of in these discussions. That said,important caveat for anyone reading this: Veridion is processing petabytes of data with predictable workloads. If you're still figuring out product-market fit or your usage patterns are all over the place, premature infrastructure optimization will absolutely kill you.The cloud's "boring reliability" has real value when you need to focus on building the actual product. The 30x multiplier is real, but it only kicks in at serious scale with stable, known workloads. Most startups aren't there yet, and that's fine.

automaticlemon 11mo ago

[dead]

as3413 11mo ago

ngl the cloud evangelists aren’t gonna like this one. this is basically “we did the math and cloud wasn’t worth it

martis2018 11mo ago

not saying they’re wrong but $2m → $60k is like… a 97% cost drop. that is either cloud pricing insanity or some very optimistic bare-metal accounting.

general1465 11mo ago
If they are going from AWS, that would be the cloud pricing insanity.

cs0min 11mo ago

can’t tell if they’re geniuses or just suffering a massive case of “we built a data center so now everything looks like a hardware nail."

florint 11mo ago

This is a super interesting analysis, I wonder what scale do you need to reach the point where the ease of adoption of AWS/GCloud is not worth it anymore. The article compares compute in each scenario, which makes sense, but I would take into account the speed to live as a secondary metric. I'm pretty sure cloud is faster, at least initially