3 ms·
I've never read such a ridiculous article. I really wanted to give them the benefit of the doubt but good lord. How is any of this simpler or better? It's like
by xyzzy_plugh 4y ago
I've never read such a ridiculous article. I really wanted to give them the benefit of the doubt but good lord. How is any of this simpler or better? It's like they prefer the pain of 2004 mixed with the worst parts of modern infrastructure. The dream of the 2000s really is alive in DHH's head, isn't it?
TFA links to their cloud spend for 2022[0], wherein lies the rub:
> In total, we spent $3,201,564 on all these cloud services in 2022. That comes out to $266,797 per month. Whew!
> For HEY, the yearly bill was $1,066,150 ($88,846/month) for production workloads only. That one service breaks down into big buckets as follows:
What the actual fuck? THREE MILLION DOLLARS? A million for their email service?? I have seen bills much larger, but for what 37signals does I am shocked. There is surely a ton of low hanging fruit to drop the bill despite the claim that it's as optimized as it can get. No way.
Even then, Hey is $99/year, and they claimed to have 25k users in the first month or so as of 2020, that's nearly $2.5MM. I presume they've grown since then. Another 2020 article[2] mentions 3/4 of their users have the iOS app, and the Android app currently shows "50k+ installs" so let's assume we're talking 200-400k users as a ceiling, ignoring attrition, which would pull $20-40MM. Even if it's half that, the cost doesn't seem unreasonable.
They're spending nearly $90k/mo on Hey. Of that the majority is RDS and OpenSearch. TFA makes it clear they know how to run MySQL, why on earth don't they stop running RDS? Both of these can easily be halved if they ran the services manually.
EKS is practically free so whatever. They state they have two deployments for ~$23k/mo total -- production is likely larger than staging but let's assume they're equal -- or ~$12k/mo each. A middle of the road EC2 instance like m4.2xlarge is less than $215/mo which gets more than enough cores and memory to run a rails app or two per node. That works out to around 55 nodes per environment. This benchmark[3] shows an m4.2xlarge can serve 172req/s via modern Ruby on Rails. At 500k users that works out to over 1600 request/user/day which seems excessive but likely within an order of magnitude of reality. These are the folks who wrote RoR so I would hope they can optimize this down further. <10000req/s for $12k/mo is pretty awful, and I'm being conservative.
Then let's talk about the $1MM/mo S3 bill. I'm not sure how to make 8PB cost that much but even the lightest touch at optimizing storage or compression or caching knocks the cost down.
This is all just nuts. There's no reason this all shouldn't be running on AWS or GKE with a much smaller bill. Their apps are predominantly CRUD, some email. Instead they replaced kubernetes with an in-house monstrosity.
I am bewildered.
[0] https://dev.37signals.com/our-cloud-spend-in-2022/ https://dev.37signals.com/our-cloud-spend-in-2022/
[1] https://www.theverge.com/2020/6/22/21298552/apple-hey-email-app-approval-rules-basecamp-launch https://www.theverge.com/2020/6/22/21298552/apple-hey-email-...
[2] https://m.signalvnoise.com/on-apples-monopoly-power-to-destroy-hey/ https://m.signalvnoise.com/on-apples-monopoly-power-to-destr...
[3] https://www.fastruby.io/blog/rails/ruby/performance/how-fast-is-ruby-3-on-rails.html https://www.fastruby.io/blog/rails/ruby/performance/how-fast...