5 ms·
Thank you for this. I'm in Europe with an established SaaS that's been running in production for years and I've converged on a similar stack (OVHCloud instead o
by znnajdla 8mo ago
Thank you for this. I'm in Europe with an established SaaS that's been running in production for years and I've converged on a similar stack (OVHCloud instead of Hetzner). However, I've realized you can stay sovereign and independent in any jurisdiction (not just Europe) just by simplifying your stack and running a few baremetal servers in-house.
Just buy a few Mac Studios and run them in-house with power supply backup and networking redundancy and you're good to go to serve more than 10k - 100k requests/second which is good enough to serve a million customers. You don't need VMs: a single Mac Studio gets you 2–4x the power of m7i.2xlarge on AWS, and pays for itself within a few months of AWS bills. You can do local AI inference and get Claude Opus-level performance (Kimi K2.5) over a cluster of Mac Studios with Exo.Labs (an unofficial Apple partner). You get free S3-compatible object storage with zero ongoing storage costs with MinIO (yes it's redundant even if you lose a server, and your hosting provider can't hold your data hostage by charging for egress). Postgres runs like a beast and is incredibly easy to setup - you get zero latency DB because it runs on the same machine, has access to lots of RAM and you're not paying per-GB or per-core. Managed databases are a scam. You don't need an Auth provider, just do passkeys yourself. And the great thing about Apple Silicon hardware is that it is amazingly quiet, reliable, and efficient - you can do thing like run headless browsers 3x faster and cheaper than on standard server hardware because of the unified memory and GPU acceleration, so you're not paying for CI/CD compute by-the-minute or headless browsers either.
This entire stack could give you computing power equivalent to a 25k euro/month AWS bill for the cost of electricity (same electricity cost as running a few fridges 24/7) plus about 50k euros one-time to set it up (about 4 Mac Studios). And yes, it's redundant, scalable, and even faster (in terms of per-request latency) than standard AWS/GCP cloud bloat. Not only is it cheaper and you own everything, but your app will work faster because all services are local (DB, Redis cache, SSD, etc.) without any VM overhead, shared cores, or noisy neighbours.
- zonkd1234 8mo agoThanks for the post. How do you currently deal with HD failures/redundancy? That’s my main concern leaving a managed database provider.
- znnajdla 8mo agoI've designed our app so that there are only two stateful services that matter: Database and Disk. Everything else is cattle, you can shut down or spin up new instances and the load balancer redirects requests with no impact. Making Postgres redundant is a matter of careful configuration with PGBouncer + HAProxy + Patroni. However for a long time we had a much simpler setup: just restore a new database from backup on a new machine if the main one failed (one-time simple script run manually - not automatic, means a little bit of downtime if there's a failure, but it worked). Or you could use CockroachDB. Making disk redundant: just use MinIO for S3-like disk (that's also where DB backups are stored). You can lose up to 2 out 4 of your servers and you lose nothing. With this setup if 1 or 2 Mac Studios fail (or need to be restarted for updates) everything just keeps running smoothly with no customer impact. It also helps that the app itself is on the Elixir BEAM (Phoenix) so everything "just works" across all machines.
- zonkd1234 8mo agocheers. Had never heard of MinIO either, very cool.
- ffsm8 8mo agoMinIO was a previously open source blob store. It's pretty old, it was basically created right around the time S3 took off. You should probably reconsider going with it in 2026 unless you're fine with their new (non -opensource) offering. It still has a "free" license, so it might still be an option depending on your priorities. But there are alternatives around, some being arguably much easier to run/maintain for small deployments like this.
- magicalhippo 8mo agoDo note MinIO is deprecated and no longer maintained, discussed here[1]. There are plenty of alternatives though, most mentioned in the referenced submission. [1]: https://news.ycombinator.com/item?id=47000041 https://news.ycombinator.com/item?id=47000041
- piltdownman 8mo agoHow do you handle anti-DDOS, zero-trust and WAF duties to a cloudflare-esque equivalency (e.g. a reverse-proxy style setup)? While I definitely concur with your conclusions re VMs and GCP hosting overhead, did you benchmark a container based setup in GKE or similar?
- znnajdla 8mo agoFor now we still use Cloudflare. Considering bunny.net after reading this OPs post.
- aa-jv 8mo agoGreat post, and interesting setup - harkens to days of old, when this was simply how things were done in the first place - but one question that I have, apropos: >.. serve more than 10k - 100k requests/second which is good enough to serve a million customers. What is your network connectivity like for this setup? Presumably you operate in a building capable of giving you fiber, with a fixed IP, or something like that?
- swiftcoder 8mo ago> Presumably you operate in a building capable of giving you fiber, with a fixed IP, or something like that? That is not really a rarity these days. I have symmetrical gigabit fibre with a fixed IP here in a Spanish farmhouse 45 minutes from the nearest population centre
- zelphirkalt 8mo agoIn some countries and with some ISPs, you cannot get a fixed IP address at all, unless you register a business and prove to the ISP that you are running a business. I am guessing they will bill you accordingly then, and still have the same shoddy connectivity. I have seen shoddy connectivity with Pyür in Germany for a whole office building. Even as a business you are not immune to bad ISPs.
- swiftcoder 8mo agoI guess Spain benefits from having a former national telecom. Movistar charges me a (outrageous by local standards) €30/month for a static IP on my residential fibre
- hvb2 8mo agoMost of those business connections come with actual SLAs though that you don't have.
- swiftcoder 8mo agoNo SLA in the world is going to help in a rural area, when a winter storm brings a tree down on the fibre :D But they offer the exact same specs to business customers in the nearby town. I appreciate Spain is well ahead of most other countries on connectivity, but I can't picture gigabit + static IP being a dealbreaker in most of Western Europe
- fauigerzigerk 8mo ago>However, I've realized you can stay sovereign and independent in any jurisdiction (not just Europe) just by simplifying your stack and running a few baremetal servers in-house. Only if you have physical offices and staff in every jurisdiction you're serving.
- znnajdla 8mo agoPresumably you have a home where you live? That's your physical office. And no you don't need a presence in every jurisdiction you serve. Visa payment network serves the world from the US.
- fauigerzigerk 8mo ago>Presumably you have a home where you live? Yes, but not where my customers live. The whole point of "sovereignty" is to serve customers from a location that is bound by the laws of _their_ jurisdiction, not mine.
- msh 8mo agoBut for that it does not matter that much where the servers are located, more where the company controlling them is located.
- fauigerzigerk 8mo agoThere are quite a few factors that matter. The place where data processing and storage takes place is one of them. It matters who can physically take control of the servers. It matters where the encryption keys are stored. The storage and processing location also matters for compliance with data residency laws. But it's not the only thing I mentioned. Having physical offices and staff in a jurisdiction usually goes along with setting up some sort of legal and taxable entity that has personally responsible directors. The whole issue is very complicated.
- dd_xplore 8mo agoI have been self hosting since couple of years, yes I got very very interested in self hosting my apps, away from the cloud overlords, but the major issue is the network. You'll need business internet plans with redundancy and based on locations that might be prohibitively expensive. Some startups might even require their own AS numbers. Also the connectivity to the data centers or cloud infra like WAF , CDNs etc will be definitely worse compared to cloud instances. Then comes firewalls, their configuration and their redundancy. These things will matter if you're serious about your SaaS.You could definitely co-locate, but that's another cost, then comes the redundancy of everything, from servers, to disks to network (routers and switches etc). I personally believe that modern hardware is pretty reliable and doesn't need redundancy in every layer, but most people won't agree with and when startups have enough money, this doesn't matter to them. I think the only reason the common public is unable to start SaaS is handling and managing these problems. Redundancy costs a lot. And many startups don't want to deal with it even if it'll help them in long run. They just gather enough cash and throw at the overlords. I do hope that the general infra should improve so that can properly host their own. Nevertheless I'm still trying to start something in SaaS space and self host from my home...
- HenriTEL 8mo agoAh yes, MinIO, that open source S3 alternative that got archived last week. To me that's the biggest problem when self-hosting services. On day to day operations, some times it just breaks and the time to get it back varies from a couple of hours to a couple of days. And for the longer term you regularly have to upgrade things yourself which takes time and energy and is stressing for stateful deployment. And then you have it, at some point maintainers are just exhausted and the project is gone.
- apexalpha 8mo agoYou can still selfhost MinIO you just have to pay. You also pay for software when renting a cloud service so this seems similar.
- tecleandor 8mo agoBut, as far as I can see in their site, the price for MinIO AIStor isn't even public, you have to "Request Pricing". And that's never a good sign.
- apexalpha 8mo agoWell MinIO has some weird quirks but I wanted to point out that "open source and free" and "self hosting" are not the same. MinIO took away the source, not the self hosting.
- tecleandor 8mo agoBut the free self hosting is only for single node, isn't it? Not fit for high availability. And the process for the rest aren't public.
- tecleandor 8mo agoWith "process" I meant "pricing".
- 8mo ago
- swiftcoder 8mo agoWhat does your networking redundancy setup look like?
- znnajdla 8mo agoGot lucky that we have a good personal relationship with our small local ISP and I trust they handle that for us. In the future I want to make it redundant by getting a second gigabit fibre connection.
- petcat 8mo ago> Just buy a few Mac Studios and run them in-house I fail to see the point of this when the system you've to decided to run "yourself" is entirely owned and dependent on another American company.
- znnajdla 8mo agoI’m not anti American, that’s not the main point of my setup. The main point is I want to own it, not rent it. Apple doesn’t control my production setup after it’s in my hands. Macs from 10 years ago still work.
- fsflover 8mo ago> Apple doesn’t control my production setup https://news.ycombinator.com/item?id=46252114 https://news.ycombinator.com/item?id=46252114
- mort96 8mo agoI imagine these Mac Minis aren't logged in to an Apple ID. Unlike Microsoft, Apple doesn't force you to connect your hardware to their cloud.
- znnajdla 8mo agoYou can setup a Mac without an Apple ID. To be honest iCloud is garbage. Almost all Mac App Store apps are available without the App Store.
- adamas 8mo agoAnd I don't see an advantage to have an Apple ID setup if you want a Mac Mini Server. All things you might need are downloable through brew.
- iririririr 8mo agoI remember having a garbge Apple id just to use xcode. Back when I was desperate enough to work from a company that only issued macbooks.
- pbronez 8mo agoAre you actually using Exo for local clustered AI inference? I’ve considered it a few times and keep finding horror stories. Never seen someone report it’s actually working well for them.
- znnajdla 8mo agoNo not yet. Planning to. But Qwen3 Coder Next 4bit runs decently well with LM Studio on my M3 Max with 96 GB RAM (50 tok/s at low context).
- ekidd 8mo ago> Managed databases are a scam. I, too, once believed this. Then I had the displeasure of watching a $10,000 server fail during Christmas travel (about 20 years ago now). A single RAID drive failed. Then, during the rebuild, a second drive failed. Then the RAID controller itself failed catastrophically, losing all the RAID volume metadata. When we restored from backup, we discovered that the sysadmin who had just quit a few weeks before had lied to us about the backup system, and we had no backups. This is the sort of black swan event that happens every 5-10 years. It's an unusually bad event, even by black swan standards, but stuff like this happens. The fundamental problem of self-hosted databases is that you test the happy path every day, but you only test true disaster recovery every 5-10 years. And in practice, this means that disaster recovery will usually fail. With a managed database service, most of what you're paying goes to making sure that disaster recovery works. And in my experience, it does. I've seen RDS database servers fail catastrophically, and completely rebuild in under 15 minutes with virtually no data loss, with almost no human intervention at all. If you care about your customers' data, I think that a reputable managed database is the right move until roughly the point that you can pay for a full time database administrator. At that point, sure, roll your own. But do regular disaster recovery tests, lest you discover that a recently departed DBA has been lying to you.
- znnajdla 8mo agoYeah but even with managed database services you don't know if your provider has invested into proper testing of their recovery so you have to test it anyway. Major services like DigitalOcean have been known to shit the bed with your backups. If you don't test your backup recovery, you don't know if you're screwed even if you're paying for "managed" services. I test my backup recovery several times a month by actually baking into our CI/CD workflow under certain conditions. The entire production database gets restored from backup every week.
- throwaway063_1 8mo agoWhat about a hybrid approach? You could use a managed db service as a live replica dedicated as a backup only. The queries would go to your local database on beefy hardware, while the replica would just have to be powerful enough to keep up with the WAL stream.
- jrmg 8mo agoYou say no VMs and are using Apple hardware. Are you running this all directly on macOS?
- znnajdla 8mo agoYes, except for one HAProxy server. The setup I described isn't fully in production yet, but my testing confirms it works. We've been running for years on one single baremetal server on Hetzner/OVH though. And macOS makes sense for one of our main workloads (headless browser agents). Much better than browser-in-linux-docker for many reasons.
- amunozo 8mo agoI have no idea how to setup something like this. How hard is to hire somebody competent enough to set a system like this in-house?
- potamic 8mo ago> You can do local AI inference and get Claude Opus-level performance (Kimi K2.5) over a cluster of Mac Studios with Exo.Labs Does it do distributed inference? What kinda token speeds do you get?
- La-Douceur 8mo agoWhen you say "baremetal servers in-house", are you talking about colocation in a european cloud provider like Hetzner, or are you talking about actually self hosting at home in your garage like some hobbyist do ? If you're talking about self hosting in your garage, I wonder how you handle networking, I mean, even if you have a good optic fiber with 1 or 10gbps, if you start getting real significant traffic, wouldn't you end up getting emails from your provider asking you why you are using 10000x more bandwith than your neighbors, and eventually be cut off ?