4 ms·
I think a lot has changed since 2013 when this article was written. Back then cloud services were less mature and there were more legitimate cases when you had
by Maro 2y ago
I think a lot has changed since 2013 when this article was written. Back then cloud services were less mature and there were more legitimate cases when you had to care about the theoretical distributed aspects of your backend architecture.. although even then these were quickly disappearing, unless you worked at a few select bigtech companies like the FAANGs.
But today, in 2024, if you just standardize on AWS, you can pretty much use one of the AWS services for pretty much anything. And that AWS service will be already distributed in the backend, for free, in terms of you not having to worry about it. Additionaly, it will be run by AWS engineers for you, with all sorts of failovers, monitoring, alerting, etc, behind the scenes that will be much better than what you can build.
So these days, for 99% of people it doesn't really make sense to worry too much about this theoretical stuff, like Paxos, Raft, consistency, vector clocks, byzantine failures, CAP, distributed locks, distributed transactions, etc. And that's good progress, it has been abstracted away behind API calls. I think it's pretty rational to just build on top of AWS (or similar) services, and accept that it's a black box distributed system that may still go down sometimes, but it'll still be 10-100x more reliable then if you try to build your own distributed system.
Of course, even for the 99%, there are still important practical things to keep in mind, like logging, debugging, backpressure, etc.
Another thing I learned is that some concepts, such as availability, are less important, and less achievable, then they seem on paper. On paper it sounds like a worthy exercise to design systems that will fail over and come back automatically if a component fails, with only a few seconds downtime. Magically, with everything working like a well oiled machine. In practice this is pretty much never works out, because there are componenets of the system that the designed didn't think of, and it's those that will fail and bring the system down. Eg. see the recent Crowdstrike incident.
And, with respect to importance of availability, of the ~10 companies I worked at in the past ~20 years there's wasn't a single one that couldn't tolerate a few hours of downtime with zero to minimal business and PR impact (people are used to things going down a couple of times a year). I remember having outages at a SaaS company I worked for 10 years ago, no revenue was coming in for a few days, but then people would just spend more in the following day. Durability is more important, but even that is less important in practice then we'd like to think [1].
The remaining 1% of engineers, who get to worry about the beuatiful theoretical AND practical aspects of distributed computing [because they work at Google or Facebook or AWS] should count themselves lucky! I think it's one of the most interesting fields in Computer Science.
I say this as somebody who deeply cares/cared about theoretical distributed computing, I wrote distributed databases [2] and papers in the past [3]. But working in industry (also managing a Platform Eng team), I cannot recall the last time I had to worry about such things.
[1] PostgreSQL used fsync incorrectly for 20 years - https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991
[2] https://github.com/scalien/scaliendb https://github.com/scalien/scaliendb
[3] https://arxiv.org/abs/1209.4187 https://arxiv.org/abs/1209.4187
- truckerbill 2y agoThe downside is you are walking into an inviting mono/duopoly. Things like this and cloudflare are amazing until the wind changes and the entire internet has a failure mode
- Maro 2y agoIn my experience, the biggest risk with AWS and similar is not the dependence itself, it's the cost. With AWS you pay a lot more for the compute then if you just rent the hardware for $25/server with Linux, and then go from there. It does make sense for some shops, where you have scale and can make calculated trade-offs between 100s of engineers' salaries and cloud costs (like Netflix). But I think a lot of small startups end up overpaying for their cloud costs even though they don't take advantage of the elasticity/scaling/replication/etc. Also, a lot of BigCos are just on public clouds because that's the thing now, but their stacks, software and teams are so broken, they don't actually take advantage of the cloud. Their engineering teams are mediocre, and so is the software, so it breaks all the time. So they end up paying for something they can't take advantage of, because they don't need to scale, they don't need high uptime and/or can't achieve it because their stuff running on top of the cloud is a horrible unreliable spaghetti anyway. If I was to do a SaaS-y startup today, I'd just rent cheap dedicated hardware, I'd use OVH [1]. I'd start with 1 reasonably beefy server for like $25-50/mo and see how far it gets me. I would use S3 for unlimited, reliable storage, (maybe even RDS for DB), but everything else I'd keep out of AWS, running on my cheap server. Ie. I'd keep my data in AWS, because that's super cheap and worth it. Later, if the thing is super successful, I'd go to $250-500/mo and get 5-10 reasonably beefy servers, and start to move things apart. I'd still not worry too much about replication and such, I'd just to backups, and take the hit if there's a problem and restore at the last backup point. I think this would get me pretty far, all the way to when I want to bring BigCo customers aboard who need all sorts of ISO standards and such. At that point the whole thing will stop being fun anyway and it's time to let other people worry about it.. And in terms of hiring, I'd make it clear at interview time that this is the stack, and we're not going to do microservices and Kubernetes and all that crap just because others are doing it, and I'd hire from the remaining pool. [1] I've been on it for my personal devboxes for ~7 years, it just works