29 ms·

Distributed systems programming has stalled

287 points by shadaj 2y ago

cruelmathlord 2y ago
it has been a while since I've seen innovation in this arena. My guess is that other domains of programming have eaten its lunch
dingnuts 2y ago
>The static-location model seems like the right place to start, since it is at least capable of expressing all the types of distributed systems we might want to implement, even if the programming model offers us little help in reasoning about the distribution. We were missing two things that the arbitrary-location model offered: > Writing logic that spans several machines right next to each other, in a single function > Surfacing semantic information on distributed behavior such as message reordering, retries, and serialization formats across network boundaries Aren't these features offered by Erlang?
- chuckledog 2y ago
  Great point. Erlang is still going strong, in fact WhatsApp is implemented in Erlang
- shadaj 2y ago
  Erlang (is great but) is still much closer to the static-location (Actors) paradigm than what I’m aspiring for. For example, if you have stateful calculations, they are typically implemented as isolated (static-location) loops that aren’t textually co-located with the message senders.
- prophesi 2y ago
  Yep, the words fault tolerance and distributed computing immediately brings to my mind Erlang/Elixir.
- deleted 2y ago
  [deleted]
bsnnkv 2y ago
Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level of burnout in my professional career. The technical aspect that contributed the most to this burnout was both the lack of observability tooling and the lack of organizational desire to invest in it. Whenever I would bring up this gap I would be told that we can't spend time/money and wait for people to create "magic tools". So far the culture in my new embedded (Rust, fwiw) position is the complete opposite. If you're burnt out working on distributed systems and you care about some of the same things that I do, it's worth giving embedded software dev a shot.
- jasonjayr 2y ago
  > Whenever I would bring up this gap I would be told that we can't spent time and wait for people to create "magic tools". That sounds like an awful organizational ethos. 30hrs to make a "magic tool" to save 300hrs across the organization sounds like a no-brainer to anyone paying attention. It sounds like they didn't even want to invest in out-sourced "magic tools" to help either.
  - bsnnkv 2y ago
    The real kicker is that it wasn't even management saying this, it was "senior" developers on the team. I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling.
    - zelphirkalt 2y ago
      Senior doesn't always mean smarter or more experienced or anything really. It just all depends on the company and its culture. It can also mean "worked for longer" (which is not equal to more experienced, as you can famously have 10 times 1y experience, instead of 10y experience) and "more aligned with how management at the company acts".
rectang 2y ago
Ten years ago, I had lunch with Patricia Shanahan, who worked for Sun on multi-core CPUs several decades ago (before taking a post-career turn volunteering at the ASF which is where I met her). There was a striking similarity between the problems that Sun had been concerned with back then and the problems of the distributed systems that power so much the world today. Some time has passed since then — and yet, most people still develop software using sequential programming models, thinking about concurrency occasionally. It is a durable paradigm. There has been no revolution of the sort that the author of this post yearns for. If "Distributed Systems Programming Has Stalled", it stalled a long time ago, and perhaps for good reasons.
- shadaj 2y ago
  Stay tuned for the next blog post for one potential answer :) My PhD has been focused on this gap!
  - rectang 2y ago
    As a programmer, I hope that your answer continues to abstract away the problems of concurrency from me, the way that CPU designers have managed, so that I can still think sequentially except when I need to. (And as a senior engineer, you need to — developing reliable concurrent systems is like pilots landing planes in bad weather, part of the job.)
    - hinkley 2y ago
      I was doing some Java code recently after spending a decade in async code and boy that first few minutes was like jumping into a cold pool. Took me a moment to switch gears back to everything is blocking and that function just takes 500ms sometimes, waiting for IO.
- EtCepeyd 2y ago
  > and perhaps for good reasons For the very good reason that the underlying math is insanely complicated and tiresome for mere practitioners (which, although I have a background in math, I openly aim to be). For example, even if you assume sequential consistency (which is an expensive assumption) in a C or C++ language multi-threaded program, reasoning about the program isn't easy. And once you consider barriers, atomics, load-acqire/store-release explicitly, the "SMP" (shared memory) proposition falls apart, and you can't avoid programming for a message passing system, with independent actors -- be those separate networked servers, or separate CPUs on a board. I claim that struggling with async messaging between independent peers as a baseline is not why most people get interested in programming. Our systems (= normal motherboards on one and, and networked peer to peer systems on the other end) have become so concurrent that doing nearly anything efficiently nowadays requires us to think about messaging between peers, and that's very-very foreign to our traditional, sequential, imperative programming languages. (It's also foreign to how most of us think.) Thus, I certainly don't want a simple (but leaky) software / programming abstraction that hides the underlying hardware complexity; instead, I want the hardware to be simple (as little internally-distributed as possible), so that the simplicity of the (sequential, imperative) programming language then reflect and match the hardware well. I think this can only be found in embedded nowadays (if at all), which is why I think many are drawn to embedded recently.
deleted 2y ago
[deleted]
tayo42 2y ago
I agreee it has stalled, I think for almost everyone what the author considers bandaid is practical enough that isn't a need for innovation. Distributed systems is more or less solved imo. Agree with another commenter, observability tools do suck. I think that's true in general for software beyond a certain amount of complexity. Storing large amounts of data for observability is expensive.
- ptmcc 2y ago
  I'm very impressed with the quality of some observability tools like Datadog, which do many good things either automatically or very easily. The usability is leaps and bounds ahead of things like New Relic or the manual instrumentation intensive open source tools. But yes, the costs are insane and require some diligence to keep from running too wild, like most SaaS products these days. But ultimately we pay it because it gives us incredibly valuable insights and has saved us countless hours in incident response, debugging, and performance profiling. It's lowered my stress level significantly.
herval 2y ago
Throwing in my two cents on the LLM impact - I've been seeing an increasing number of systems where core part of the functionality is either LLMs or LLM-generated code (sometimes on the fly, sometimes cached for reuse). If you think distributed systems were difficult before, try to imagine a system where the code being executed _isn't even debuggable or repeatable_. It feels like we're racing towards a level of complexity in software that's just impossible for humans to grasp.
- klysm 2y ago
  That's okay though! We can just make LLMs grasp it!
  - moffkalast 2y ago
    "There's always a larger model."
  - herval 2y ago
    ironically or not, the best way to have LLMs be effective at writing valid code is when they work on microservices. Since the scope is smaller and the boundary is clear, tools like Cursor/Windsurf seem to make very few mistakes (compared to pointing them at your monorepo, where they usually end up completely wrong)
    - klysm 2y ago
      Is it then up to the human to specify the services and how they interact?
hinkley 2y ago
I am so appalled every time I ask a group of devs and get the same answer that I’ve just stopped asking. How many of you took a distributed programming class in college? And it turns out yet again that not only am I the only one, but that none of them recollect it even being in the course catalog. For me it was a required elective (you must take at least one of these 2-3 classes). And I went to college while web browsers were being invented. When Cloud this and Cloud that started every university should have added it to the program. What the fuck is going on with colleges?
- klysm 2y ago
  Definitely a problem, but I think there's always a gap here. What are colleges optimizing for with computer science programs? I would wager there is an incentive problem at the core which is causing these gaps to occur.
- shermantanktop 2y ago
  My .02 is that any topic sufficiently important shouldn't be left to colleges. You can force feed a complex topic to a bunch of undergrads, but they will forget 95% of it, and 5 years later they'll say "ohhh, I think I have a textbook on that in my parents' basement." The reality is that most of this profession is learned on the job, and college acts as a filter at the start of the funnel. If someone is not capable of picking up the Paxos paper, then having had someone tell them about it 5 years ago when they had a hangover won't help.
  - hinkley 2y ago
    I am 100% convinced we could delete the compiler class from college curricula and replace it with distributed computing and the world would be a better place.
    - bigmutant 2y ago
      Def agree. Most people will never touch an Abstract Syntax Tree or even Expression Trees. Almost everyone working in back-end will use Cloud Services, will make mistakes based on assumptions of what they provide
gklitt 2y ago
This is outside my area of expertise, but the post sounds like it’s asking for “choreographic programming”, where you can write an algorithm in a single function while reasoning explicitly about how it gets distributed: https://en.m.wikipedia.org/wiki/Choreographic_programming https://en.m.wikipedia.org/wiki/Choreographic_programming I’m curious to what extent the work in that area meets the need.
- shadaj 2y ago
  You caught me! That's what my next post is about :)
  - ashton314 2y ago
    I see from your bio that you are a PhD student. What are you doing with choreographies? (I’m in this space too.)
  - lachlan_gray 2y ago
    This could be a fun example to work with :p https://en.m.wikipedia.org/wiki/Shakespeare_Programming_Language https://en.m.wikipedia.org/wiki/Shakespeare_Programming_Lang...
    - shadaj 2y ago
      You might enjoy my first ever blog post from ~10 years ago, when I first learned about distributed systems: https://www.shadaj.me/writing/romeo-juliet-and-reactive-programming https://www.shadaj.me/writing/romeo-juliet-and-reactive-prog...
- roadbuster 2y ago
  How does "choreographic programming" differ from the actor model?
  - LegionMammal978 2y ago
    From what I can tell, the important distinction is that all actors (and their messages) are described alongside each other, instead of being described separately. There are many implementations of the actor model, but most of them are the 'static-location architectures' that TFA talks about.
cmrdporcupine 2y ago
Two things: Distributed systems are difficult to reason about. Computer hardware today is very powerful. There is a yo-yo process in our industry over the last 50 years between centralization and distribution. We necessarily distribute when we hit the limits of what centralization can accomplish because in general centralization is easier to reason about. When we hit those junctures, there's a flush of effort into distributed systems. The last major example of this I can think of was the 2000-2010 period, when MapReduce, "NoSQL" databases, Google's massive arrays of supposedly identical commodity grey boxes (not the case anymore), the High Scalability blog, etc. were the flavour of the time. But then, frankly, mass adoption of SSDs, much more powerful computers, etc. made a lot of those things less necessary. The stuff that most people are doing doesn't require a high level of distributed systems sophistication. Distributed systems are an interesting intellectual puzzle. But they should be a means to an end not an end in themselves.
- porridgeraisin 2y ago
  I think the reason that distributed systems still are the go-to choice for many software teams is to do with people/career expectations/careers orienting themselves around distributed systems over the time period you mentioned. It will take a while for it to re-orient, and then distributed systems might become a fad again ;) An example of this is typical promotion incentives being easier to get in microservice teams, thereby incentivising people to organize the team/architecture in that way.
  - cmrdporcupine 2y ago
    Honestly, I am more cynical and just think people are always looking for ways to make their jobs more interesting than they actually are.
    - Nevermark 2y ago
      > Honestly, I am more cynical and just think people are always looking for ways to make their jobs more interesting than they actually are. When you frame the problem that way, unnecessary complexity seems like part of a healthy solution path. /h Companies get reliability benefits from slack, but creative people abhor wasted slack. Some basic business strategy/wisdom for maintaining/managing creative slack is needed.
taeric 2y ago
I'm not clear what the proposal here is? It specifically eschews tooling as a driver in the solution, but why? Wouldn't tooling be one of the most likely areas to get solid progress, as you could make tooling and point it at existing products. Would be interesting to see comparisons to other domains. Surely you could look at things like water processing plants to see how they build and maintain massive structures that do coordinated work between parts of it? Power generation plants. Assembly factories. Do we not have good artifacts for how these things are designed and reasoned about?
sriram_malhar 2y ago
The X10 language captures the notion of location of computation as a language primitive ("place") http://x10-lang.org http://x10-lang.org
tracnar 2y ago
The unison programming language does foray into a truly distributed programming language: https://www.unison-lang.org/ https://www.unison-lang.org/
- aDyslecticCrow 2y ago
  Functional programming languages already have alot of powerful concepts for distributed programming. Loads of the distributed programming techniques used elsewhere are often taken from an obscure fp language from years prior. Erlang comes to mind as still quite uniquely distributed without non fp comparison Unison seems to build on it further. Very cool
hinkley 2y ago
I don’t think there’s anyone in the Elixir community who wouldn’t love it if companies would figure out that everyone is writing software that contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang, and start hiring Elixir or Gleam devs. The future is here, but it is not evenly distributed.
- deleted 2y ago
  [deleted]
- ikety 2y ago
  It's so odd seeing people dissuade others for implementing "niche" languages like Elixir or Gleam. If you post a job opportunity with these languages, I guarantee you will be swamped with qualified candidates that are very passionate and excited to work with these languages full time.
  - hinkley 2y ago
    At this point I’m worried that because elixir is over 10 years old that it’ll never arrive. But then Python is older than Java and here we are.
- ignoramous 2y ago
  > writing software that contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang Since you were at AWS (?), you'd know that Erlang did get its shot at distributed systems there. I'm unsure what went wrong, but if not c/c++, it was all JVM based languages soon after that.
  - hinkley 2y ago
    No I worked a contract in the retail side and would not wish that job on anyone. My most recent favorite boss works there now and I haven’t even said hi because I’m afraid he’ll offer me a job.
  - worthless-trash 2y ago
margorczynski 2y ago
Distributed systems are cool but most people don't really get how much complexity it introduces which leads them to fad-driven decisions like using Event Sourcing where there is no fundamental need to use it. I've seen projects getting burned because of the complexity and overhead it introduces where "simpler" approaches worked well and were easy to extend/fix. Hard to find and fix bugs, much slower feature addition and lots of more goodies the blogs with toy examples don't speak about.
- nine_k 2y ago
  The best recipe I know is to start from a modular monolith [1] and split it when and if you need to scale way past a few dozen nodes. Event sourcing is a logical structure; you can implement it with SQLite or even flat files, locally, if you your problem domain is served well by it. Adding Kafka as the first step is most likely a costly overkill. [1]: https://awesome-architecture.com/modular-monolith/ https://awesome-architecture.com/modular-monolith/
  - margorczynski 2y ago
    What you're speaking of is a need/usability-based design and extension where you design the solution with certain "safety valves" that let you scale it up when needed. This is in contrast to the fad-driven design and over-engineering that I'm speaking of (here I simply used ES as an example) that is usually introduced because someone in power saw a blog post or 1h talk and it looked cool. And Kafka will be used because it is the most "scalable" and shiny solution, there is no pros-vs-cons analysis.
- rjbwork 2y ago
  If the choice has already been made to do a distributed system (outside of the engineer's control...), is a choice to use Event Sourcing by the engineer then a good idea?
- mrkeen 2y ago
  In my experience: 1) We are surrounded by distributed systems all the time. When we buy and sell B2B software, we don't know what's stored in our partners databases, they don't know what's in ours. Who should ask whom, and when? If the data sources disagree, whose is correct? Just being given access to a REST API and a couple of webhooks is all you need to be in full distributed systems land. 2) I honestly do not know of a better approach than event-sourcing (i.e. replicated state machine) to coordinate among multiple masters like this. The only technique I can think of that comes close is Paxos - which does not depend on events. But then the first thing I would do if I only had Paxos, would be to use it to bootstrap some kind of event system on top of it. Even the non-event-sourcing technologies like DBs use events (journals, write-ahead-logs, sstables, etc.) in their own implementation. (However that does not imply that you're getting events 'for free' by using these systems.) My co-workers do not put any alternatives forward. Reading a database, deciding what action to do, and then carrying out said action is basically the working definition of a race-condition. Bankers and accountants had this figured out thousands of years ago: a bank can't send a wagon across the country with queries like "How much money is in Joe's account?" wait a week for the reply, and then send a second wagon saying "Update Joe's account so it has $36.43 in it now". It's laughable. But now that we have 50-150ms latencies, we feel comfortable doing GETs and POSTs (with a million times more traffic) and somehow think we're not going to get our numbers wrong. Like, what's an alternative? I have a shiny billion-dollar fully-ACID SQL db with my customer accounts in them. And my SAAS partner bank also has that technology. Put forward literally any idea other than events that will let us coordinate their accounts such that they're not able to double-spend money, or are prevented from spending money if a node is down. I want an alternative to event sourcing.
mgraczyk 2y ago
The author is missing information about LLMs. In the "Obligatory LLM Section" he focuses on distributed systems that use LLMs. But almost all of the new innovation I'm familiar with in distributed systems is about training LLMs. I wouldn't say the programming techniques are "new" in the way this post is describing them, but the specifics are pretty different from building a database or data pipeline engine (less message oriented, more heavily pipelined, more low level programming, etc)
- riku_iki 2y ago
  > But almost all of the new innovation I'm familiar with in distributed systems is about training LLMs I think database space is still hot topic with many unsolved problems.
sanity 2y ago
The article makes great points about why distributed programming has stalled, but I think there's still room for innovation—especially in how we handle state consistency in decentralized systems. In Freenet[1], we’ve been exploring a novel approach to consistency that avoids the usual trade-offs between strong consistency and availability. Instead of treating state as a single evolving object, we model updates as summarizable deltas—each a commutative monoid—allowing peers to merge state independently in any order while achieving eventual consistency. This eliminates the need for heavyweight consensus protocols while still ensuring nodes converge on a consistent view of the data. More details here: https://freenet.org/news/summary-delta-sync/ https://freenet.org/news/summary-delta-sync/ Would love to hear thoughts from others working on similar problems! [1] https://freenet.org/ https://freenet.org/
- Karrot_Kream 2y ago
  Haven't read the post yet (I should, I have been vaguely following y'all along but obviously not close enough!) How is this different from delta-based CRDTs? I've built (admittedly toy) CRDTs as DAGs that ship deltas using lattice operations and it's really not that hard to have it work. There's already CRDT based distributed stores out there. How is this any different?
  - sanity 2y ago
    Good question! Freenet is a decentralized key-value store, but unlike traditional KV stores, the keys are WebAssembly (WASM) contracts. These contracts define not just what values (i.e., data or state) are valid for that key but also when and how the value can be mutated. They also specify how to efficiently synchronize the value across peers using summaries and deltas. Each contract determines how state changes are validated, summarized, and merged, meaning you can efficiently implement almost any CRDT mechanism in WASM on top of Freenet. Another key difference is that Freenet is an observable KV store, allowing you to subscribe to values and receive immediate updates when they change.
    - Karrot_Kream 2y ago
MisterTea 2y ago
This has been an issue for quite some time. Rob Pike wrote about it going back - https://doc.cat-v.org/bell_labs/utah2000/ https://doc.cat-v.org/bell_labs/utah2000/
bigmutant 2y ago
Good resources for understanding Distributed Systems: - MIT course with Robert Morris (of Morris Worm fame): https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39... - Martin Kleppmann (author of DDIA): https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjc... If you can work through the above (and DDIA), you'll have a solid understanding of the issues in Distributed System, like Consensus, Causality, Split Brain, etc. You'll also gain a critical eye of Cloud Services and be able to articulate their drawbacks (ex: did you know that replication to DynamoDB Secondary Indexes is eventually consistent? What effects can that have on your applications?)
- ignoramous 2y ago
  > Robert Morris (of Morris Worm fame) (of Y Combinator fame, too)
Karrot_Kream 2y ago
When I was graduating from my Masters (failed PhD :) this overview of various programming models is generally how I thought of things. I've been writing distributed code now in industry for a long time and in practice, having worked at a some pretty high-scale tech companies over the years, most shops tend to favor static-location style models. As the post states, it's due largely to control and performance. Scaling external-distribution systems has been difficult everywhere I've seen it tried and usually ends up creating a few knowledgeable owners of a system with high bus-factor. Scaling tends to work fine until it doesn't and these discontinuous, sharp edges are very very painful as they're hard to predict and allocate resourcing for. Are external-distribution systems dead ends then? Even if they can achieve high theoretical performance, operation of these systems tends to be very difficult. Another problem I find with external-distribution systems is that there's a lot of hidden complexity in just connecting, reading, and writing to them. So you want to talk to a distributed relational DB, okay, but are you using a threaded concurrency model or an async concurrency model? You probably want a connection pool so that TCP HOL blocking doesn't tank your throughput. But if you're using threads, how do you map your threads to the connections in the pool? The pool itself represents a bottleneck as well. How do you monitor the status of this pool? Tools like Istio strive to standardize this a little bit but fundamentally we're working with 3 domains here just to write to the external-distribution system itself: the runtime/language's concurrency model, the underlying RPC stack, and the ingress point for the external-distribution system. Does anyone have strong stories of scaling an external-distribution system that worked well? I'd be very curious. I agree that progress here has stalled significantly. But I find myself designing big distributed architecture after big distributed architecture continuing to use my deep experience of architecting these systems to build static-location systems because if I'm already dealing with scaling pains and cross-domain concerns, I may as well rip off the band-aid and be explicit about crossing execution domains.
KaiserPro 2y ago
Distributed systems are hard, as well all know. However the number of people that actually need a distributed system is pretty small. With the rise of kubernetes, the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped. You go distributed either because you are desperate, or because you think it would be fun. K8s takes the fun out of most things. Moreover, with machines suddenly getting vast IO improvements, the need for going distributed is much less than it was 10 years. (yes i know there is fault tolerance, but that adds another dimension of pain.)
- sd9 2y ago
  > the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped Gosh, this was hard to parse! I’m still not sure I’ve got it. Do you mean “kubernetes has caused more people to suffer due to going distributed unnecessarily”, or something else?
  - boarush 2y ago
    Had me confused for a second too, but I think it is the former that they meant. K8s has unneeded complexity which is really not required at even decent enough scales, if you've put in enough effort to architect a solution that makes the right calls for your business.
    - KaiserPro 2y ago
      yeah sorry, double negatives. People got burnt by kubernetes, and that pissed in the well of enthusiasm for experimenting with distributed systems
      - DrFalkyn 2y ago
        Because people, especially Devops, thought k8s was some magic, when all it really does is makes the mechanics easier If you’re architecture is poor k8s won’t help you
- 2y ago
synergy20 2y ago
I saw comments about embedded development, which I have been doing that for a long time, just want to make a point here: the pay has upper limits, you will be paid fine but will reach the pay limit very fast, and it will stay there for the rest of your career. they can swap someone in with that price tag to do whatever you are working on, because, after all, embedded devel is not rocket science.
- cmrdporcupine 2y ago
  The problem with embedded is its proximity to EE which is frankly underpaid. But it's also more that the "other" kind of SWE work -- "backend" etc is frankly overpaid because of the copious quantities of $$ dumped into it by VC and ad money.
sn9 2y ago
It sounds like they're looking for the Rust project Hydro [0]. EDIT: Lol nvm the author is one of the authors of Hydro [1]. [0] https://github.com/hydro-project/hydro https://github.com/hydro-project/hydro [1] https://hydro.run/people https://hydro.run/people
- seivan 2y ago
  [dead]
jderick 2y ago
Distributed systems are hard. I like the idea of "semantic locality." I think it can be achieved to some degree via abstraction. The code that runs across many machines does a lot of stuff but only a small fraction of that is actually involved in coordination. If you can abstract away those details you should end up with a much simpler protocol that can be modeled in a succinct way. Then you can verify your protocol much more easily. Formal methods have used tools such as spin (promela) or guarded commands (murphi) for modeling these kinds of systems. I'm sure you could do something similar with the lean theorem prover. The tricky part is mapping back and forth between your abstract system and the real one. Perhaps LLMs could help here. I work on hardware and concurrency is a constant problem even at that low level. We use model checking tools which can help.
lifeisstillgood 2y ago
This is a massive coming issue - I am not sure “distributed” can be exactly replaced with “parallel processing” but it’s close So to simplify, from 1985 to 2005 ish you could keep sequential software exactly the same and it just ran faster each new hardware generation. One CPU but transistors got smaller and (hand wavy, on chip ram, pipelining ) Then roughly around 2010 single CPUs just stopped magically doubling. You got more cores, but that meant parallel or distributed programming - your software that in 1995 served 100 people was the same serving 10,000 people in 2000. But in 2015 we needed new coding - we got NOSQL and map reduce and facebook data centres. But the hardware kept growing TSMC now has wafer scale chips with 900,000 cores - but my non parallel, on distributed code won’t run 1 million times faster - Amdahls law just won’t let me So yeah - no one wants to buy new chips with a million cores because you aren’t going to get the speed ups - why buy an expensive data centre full of 100x cores if you can’t sell them at 100x usage.
kodablah 2y ago
> Just like the external-distribution model, arbitrary-location architectures often come with a performance cost. Durable execution systems typically snapshot their state to a persistent store between every step. This is not true by most definitions of "snapshot". Most (all?) durable execution systems use event sourcing and therefore it's effectively an immutable event log. And it's only events that have external side effects enough to rebuild the state, not all state. While technically this is not free, it's much more optimal than the traditional definition of capturing and storing a "snapshot". > But this simplicity comes at a significant cost: control. By letting the runtime decide how the code is distributed [...] we don’t want to give up: Explicit control over placement of logic on machines, with the ability to perform local, atomic computations Not all durable execution systems require you to give this up completely. Temporal (disclaimer: my employer) allows grouping of logical work by task queue which many users use to pick locations of work, even so far as a task queue per physical resource which is very common for those wanting that explicit control. Also there are primitives for executing short, local operations within workflows assuming that's what is meant there.
anonymousDan 2y ago
This article is just word salad. In what way does Redis 'abstract distribution' for example?
gregw2 2y ago
What I noticed missing in this analysis of distributed systems programming was a recognition/discussion of how distributed databases (or datalakes) decoupling storage from compute have changed the art of the possible. In the old days of databases, if you put all your data in one place, you could scale up (SMP) but scaling out (MPP) really was challenging. Nowdays, you (iceberg), or a DB vendor (Snowflake, Databricks, BigQuery, even BigTable, etc), put all your data on S3/GCS/ADLS and you can scale out compute to read traffic as much as you want (as long as you accept something like a snapshot isolation read level and traffic is largely read-only or writes are distributed across your tables and not all to one big table.) You can now share data across your different compute nodes or applications/systems by managing permissions pointers managed via a cloud metadata/catalog service. You can get microservice databases without each having completely separate datastores in a way.
thway15269037 2y ago
Oh god, even this article has AI and LLM section in it. When I thought distributed system design could not get any worse, someone actually pitched AI slop in it. God I want to dig a cave and live in it.
- Nevermark 2y ago
  > When I thought distributed system design could not get any worse, someone actually pitched AI slop in it. I am not sure that pointing out that today's models are going to be MUCH worse at reasoning about distributed code than serial code is "pitching". Conversely, pointing out that the reason they are so bad at distributed is the lack of related information locality, the same problem humans often have, puts a reasonable second underline on the value of more locality in our development artifacts.
hintymad 2y ago
This reminds me of Rob Pike's article "Systems Software Research is Irrelevant," written about 15 years ago. Perhaps many systems have matured to a point where any improvement appears incremental to engineers, so the conviction to develop a new programming model isn't strong enough. Or perhaps we're in a temporary plateau, and a groundbreaking tool will emerge in a few years. Regarding Laddad's point, building tools native to distributed systems programming might be intrinsically difficult. It's not for lack of trying. We've invented numerous algebras, calculi, programming models, and experimental programming languages over the past decades, yet somehow none has really taken off. If anything, I'd venture to assert that object storage, perhaps including Amazon DynamoDB, has changed the landscape of programming distributed systems. These two systems, which optimize for throughput and reliability, make programming distributed systems much easier. Want a queue system? Build on top of S3. Want a database? Focus on query engines and outsource storage to S3. Want a task queue? Just poll DDB tables. Want to exchange states en masse? Use S3. The list goes on. Internally to S3, I think the biggest achievement is that S3 can use scalability to its advantage. Adding a new machine makes S3 cheaper, faster, and more reliable. Unfortunately, this involves multiple moving parts and is therefore difficult to abstract into a tool. Perhaps an arbitrarily scalable metadata service is what everyone could benefit from? Case in point, Meta's warm storage can scale to multiple exabytes with a flat namespace. Reading the paper, I realized that many designs in the warm storage are standard, and the real magic lies in its metadata management, which happens to be outsourced to Meta's ZippyDB. Meanwhile, open-source solutions often boast about their scalability, but in reality, all known ones have certain limits, usually no more than 100PBs or a few thousand nodes.
- jbarham 2y ago
  > This reminds me of Rob Pike's article "Systems Software Research is Irrelevant," written about 15 years ago. 25 years ago: http://herpolhode.com/rob/utah2000.pdf http://herpolhode.com/rob/utah2000.pdf (Time flies.)
  - hintymad 2y ago
    Wow, time flies. Thanks!
nyrikki 2y ago
> Distributed SQL Engines This is what I see holding some applications back. The relational model is flexible and sufficient for many needs but the ACID model is responsible for much of the complexity in some more recent solutions. While only usable for one-to-many relationships, the hierarchical model would significantly help in some of the common areas like financial transactions. Think IBM IMS fastpath, and the related channel model. But it seems every neo paradime either initially hampers itself, or grows to be constrained by Codd's normalization rules, which result in transitive closure a the cost of independence. As we have examples like Ceph's radios, Kafka etc...if you view the hierarchical file path model as being intrinsic to that parent child relationship we could be distributed. Perhaps materialized views could be leveraged to allow for SQL queries without turning the fast path into a distributed monolith. SQL is a multi tool, and sometimes you just need to use a specific tool.
th0ma5 2y ago
Since multicore processing a ton of software you use or create is distributed you have to ask if you want to be in control of how it is distributed or not. If you want it easy and let the library figure it out then you have to accept the topological ideas it has. For instance H2O is a great machine learning package that even has its own transparent multi core processing. If you want to go across machines it has its own cluster built in. You can also install it into Hadoop, Spark, etc but once you start going that direction you're more and more on the hook for what that means and if it even is more effective for your problem and what your distributed strategy should be. Things like re-entrant idempotence, software transactional memory, copy on write, CRDTs etc are going to have waste and overhead but can vastly simplify conceptually the ongoing development and maintenance of even non-distributed efforts in my opinion, and we keep having the room to eat the overhead. There's a ton of bias against this for good reasons that the non distributed concepts still just work without any hassle but we'd be less in the mud in a fundamental way of we learned to let go of non-eventual consistency.
samonurofil 2y ago
[dead]
rstuart4133 2y ago
> Although static-location architectures offer developers the most low-level control over their system, in practice they are difficult to implement robustly without distributed systems expertise. This is the understatement of the article. There are two insanely difficult things to get right in computers. One is cryptography, and other is distributed systems. I'd argue the latter is harder. The reason simple enough to understand. In any program the programmer has to carry in his head every piece of state that is accessible at any given point, the invariants that apply to that state, and the code responsible for modifying that state while preserving the invariants. In sequential programs the code that can modify the shared state is restricted to inner loops and functions you call, and you have to verify every modification preserves the invariants. It's a lot. The hidden enemy is aliasing, and you'll find entire books written on the counter measures like immutable objects, function programming, and locks. Coordinating all this is so hard only a small percentage of the population can program large systems. I guess you are thinking "but of a lot of people here can do that". True, but we are a tiny percentage. In distributed systems those blessed restrictions a single execution thread gives us on what code can access shared state goes out the window. Every line that could read or write the shared state has to be considered, whether its adjacent or not, whether you called it here or not. The state interactions explode in the same way interactions between qubits explode. Both explode beyond the capability of human minds to assemble them all in one place. You have to start forming theorems and formulating proofs. That worst part is newbie programmers are not usually aware this explosion has taken place. That's why experienced software engineers give the following advice on threads: just don't. You don't have a feel for what will happen, your code will appear to work when you test it while being rabbit warren of disastrous bugs that will likely never be fixed. It's why Linux RCU author Paul McKenney is still not confident his code is correct, despite being one of the greatest concurrent programming minds on the planet. It's why Paxos is hard to understand despite being relatively simple. Expecting an above average programmer to work on a distributed system and not introduce bugs without leaning on one of one of the "but it is inefficient" tools he lists is an impossible dream. A merely experienced average has no hope. It's hard. Only a tiny, tiny fraction of the programmers on the planet can pull it off kind of hard.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
sakesun 2y ago
I have been told and believed that we are getting very close to the ultimate answer throughout my career since CORBA/DCOM. Just learn that there is another discontinued attempt https://serviceweaver.dev/ https://serviceweaver.dev/
nchammas 2y ago
There is an old project out of Berkeley called BOOM [1] that developed a language for distributed programming called Bloom [2]. I don't know enough about it to map it to the author's distributed programming paradigms, but the Bloom features page [3] is interesting: > disorderly programming: Traditional languages like Java and C are based on the von Neumann model, where a program counter steps through individual instructions in order. Distributed systems don’t work like that. Much of the pain in traditional distributed programming comes from this mismatch: programmers are expected to bridge from an ordered programming model into a disordered reality that executes their code. Bloom was designed to match–and exploit–the disorderly reality of distributed systems. Bloom programmers write programs made up of unordered collections of statements, and are given constructs to impose order when needed. [1]: https://boom.cs.berkeley.edu https://boom.cs.berkeley.edu [2]: http://bloom-lang.net/index.html http://bloom-lang.net/index.html [3]: http://bloom-lang.net/features/ http://bloom-lang.net/features/
- jmhucb 2y ago
  Good pattern matching. Bloom is a predecessor project to the OP's PhD thesis work :-) This area takes time and many good ideas to mature, but as the post hints, progress is being made.
ConanRus 2y ago
No Erlang mention? Sad.
anacrolix 2y ago
I've been trying to explain this to people for 8 years. All of our existing languages side step the problem. Developers are writing distributed systems every day but seem oblivious to the fact their tools aren't helping at all.
hackburg 2y ago
[dead]
lincpa 2y ago
[dead]

287 points by shadaj 2y ago

it has been a while since I've seen innovation in this arena. My guess is that other domains of programming have eaten its lunch

dingnuts 2y ago

>The static-location model seems like the right place to start, since it is at least capable of expressing all the types of distributed systems we might want to implement, even if the programming model offers us little help in reasoning about the distribution. We were missing two things that the arbitrary-location model offered: > Writing logic that spans several machines right next to each other, in a single function > Surfacing semantic information on distributed behavior such as message reordering, retries, and serialization formats across network boundaries Aren't these features offered by Erlang?

chuckledog 2y ago
Great point. Erlang is still going strong, in fact WhatsApp is implemented in Erlang
shadaj 2y ago
Erlang (is great but) is still much closer to the static-location (Actors) paradigm than what I’m aspiring for. For example, if you have stateful calculations, they are typically implemented as isolated (static-location) loops that aren’t textually co-located with the message senders.
prophesi 2y ago
Yep, the words fault tolerance and distributed computing immediately brings to my mind Erlang/Elixir.
deleted 2y ago
[deleted]

bsnnkv 2y ago

Last month I switched from a role working on a distributed system (FAANG) to a role working on embedded software which runs on cards in data center racks. I was in my last role for a year, and 90%+ of my time was spent investigating things that went "missing" at one of many failure points between one of the many distributed components. I wrote less than 200 lines of code that year and I experienced the highest level of burnout in my professional career. The technical aspect that contributed the most to this burnout was both the lack of observability tooling and the lack of organizational desire to invest in it. Whenever I would bring up this gap I would be told that we can't spend time/money and wait for people to create "magic tools". So far the culture in my new embedded (Rust, fwiw) position is the complete opposite. If you're burnt out working on distributed systems and you care about some of the same things that I do, it's worth giving embedded software dev a shot.

jasonjayr 2y ago
> Whenever I would bring up this gap I would be told that we can't spent time and wait for people to create "magic tools". That sounds like an awful organizational ethos. 30hrs to make a "magic tool" to save 300hrs across the organization sounds like a no-brainer to anyone paying attention. It sounds like they didn't even want to invest in out-sourced "magic tools" to help either.
- bsnnkv 2y ago
  The real kicker is that it wasn't even management saying this, it was "senior" developers on the team. I wonder if these roles tend to attract people who get the most job enjoyment and satisfaction out of the (manual) investigation aspect; it might explain some of the reluctance to adopting or creating more sophisticated observability tooling.
  - zelphirkalt 2y ago
    Senior doesn't always mean smarter or more experienced or anything really. It just all depends on the company and its culture. It can also mean "worked for longer" (which is not equal to more experienced, as you can famously have 10 times 1y experience, instead of 10y experience) and "more aligned with how management at the company acts".

rectang 2y ago

Ten years ago, I had lunch with Patricia Shanahan, who worked for Sun on multi-core CPUs several decades ago (before taking a post-career turn volunteering at the ASF which is where I met her). There was a striking similarity between the problems that Sun had been concerned with back then and the problems of the distributed systems that power so much the world today. Some time has passed since then — and yet, most people still develop software using sequential programming models, thinking about concurrency occasionally. It is a durable paradigm. There has been no revolution of the sort that the author of this post yearns for. If "Distributed Systems Programming Has Stalled", it stalled a long time ago, and perhaps for good reasons.

shadaj 2y ago
Stay tuned for the next blog post for one potential answer :) My PhD has been focused on this gap!
- rectang 2y ago
  As a programmer, I hope that your answer continues to abstract away the problems of concurrency from me, the way that CPU designers have managed, so that I can still think sequentially except when I need to. (And as a senior engineer, you need to — developing reliable concurrent systems is like pilots landing planes in bad weather, part of the job.)
  - hinkley 2y ago
    I was doing some Java code recently after spending a decade in async code and boy that first few minutes was like jumping into a cold pool. Took me a moment to switch gears back to everything is blocking and that function just takes 500ms sometimes, waiting for IO.
EtCepeyd 2y ago
> and perhaps for good reasons For the very good reason that the underlying math is insanely complicated and tiresome for mere practitioners (which, although I have a background in math, I openly aim to be). For example, even if you assume sequential consistency (which is an expensive assumption) in a C or C++ language multi-threaded program, reasoning about the program isn't easy. And once you consider barriers, atomics, load-acqire/store-release explicitly, the "SMP" (shared memory) proposition falls apart, and you can't avoid programming for a message passing system, with independent actors -- be those separate networked servers, or separate CPUs on a board. I claim that struggling with async messaging between independent peers as a baseline is not why most people get interested in programming. Our systems (= normal motherboards on one and, and networked peer to peer systems on the other end) have become so concurrent that doing nearly anything efficiently nowadays requires us to think about messaging between peers, and that's very-very foreign to our traditional, sequential, imperative programming languages. (It's also foreign to how most of us think.) Thus, I certainly don't want a simple (but leaky) software / programming abstraction that hides the underlying hardware complexity; instead, I want the hardware to be simple (as little internally-distributed as possible), so that the simplicity of the (sequential, imperative) programming language then reflect and match the hardware well. I think this can only be found in embedded nowadays (if at all), which is why I think many are drawn to embedded recently.

deleted 2y ago

[deleted]

tayo42 2y ago

I agreee it has stalled, I think for almost everyone what the author considers bandaid is practical enough that isn't a need for innovation. Distributed systems is more or less solved imo. Agree with another commenter, observability tools do suck. I think that's true in general for software beyond a certain amount of complexity. Storing large amounts of data for observability is expensive.

ptmcc 2y ago
I'm very impressed with the quality of some observability tools like Datadog, which do many good things either automatically or very easily. The usability is leaps and bounds ahead of things like New Relic or the manual instrumentation intensive open source tools. But yes, the costs are insane and require some diligence to keep from running too wild, like most SaaS products these days. But ultimately we pay it because it gives us incredibly valuable insights and has saved us countless hours in incident response, debugging, and performance profiling. It's lowered my stress level significantly.

herval 2y ago

Throwing in my two cents on the LLM impact - I've been seeing an increasing number of systems where core part of the functionality is either LLMs or LLM-generated code (sometimes on the fly, sometimes cached for reuse). If you think distributed systems were difficult before, try to imagine a system where the code being executed _isn't even debuggable or repeatable_. It feels like we're racing towards a level of complexity in software that's just impossible for humans to grasp.

klysm 2y ago
That's okay though! We can just make LLMs grasp it!
- moffkalast 2y ago
  "There's always a larger model."
- herval 2y ago
  ironically or not, the best way to have LLMs be effective at writing valid code is when they work on microservices. Since the scope is smaller and the boundary is clear, tools like Cursor/Windsurf seem to make very few mistakes (compared to pointing them at your monorepo, where they usually end up completely wrong)
  - klysm 2y ago
    Is it then up to the human to specify the services and how they interact?

hinkley 2y ago

I am so appalled every time I ask a group of devs and get the same answer that I’ve just stopped asking. How many of you took a distributed programming class in college? And it turns out yet again that not only am I the only one, but that none of them recollect it even being in the course catalog. For me it was a required elective (you must take at least one of these 2-3 classes). And I went to college while web browsers were being invented. When Cloud this and Cloud that started every university should have added it to the program. What the fuck is going on with colleges?

klysm 2y ago
Definitely a problem, but I think there's always a gap here. What are colleges optimizing for with computer science programs? I would wager there is an incentive problem at the core which is causing these gaps to occur.
shermantanktop 2y ago
My .02 is that any topic sufficiently important shouldn't be left to colleges. You can force feed a complex topic to a bunch of undergrads, but they will forget 95% of it, and 5 years later they'll say "ohhh, I think I have a textbook on that in my parents' basement." The reality is that most of this profession is learned on the job, and college acts as a filter at the start of the funnel. If someone is not capable of picking up the Paxos paper, then having had someone tell them about it 5 years ago when they had a hangover won't help.
- hinkley 2y ago
  I am 100% convinced we could delete the compiler class from college curricula and replace it with distributed computing and the world would be a better place.
  - bigmutant 2y ago
    Def agree. Most people will never touch an Abstract Syntax Tree or even Expression Trees. Almost everyone working in back-end will use Cloud Services, will make mistakes based on assumptions of what they provide

gklitt 2y ago

This is outside my area of expertise, but the post sounds like it’s asking for “choreographic programming”, where you can write an algorithm in a single function while reasoning explicitly about how it gets distributed: https://en.m.wikipedia.org/wiki/Choreographic_programming https://en.m.wikipedia.org/wiki/Choreographic_programming I’m curious to what extent the work in that area meets the need.

shadaj 2y ago
You caught me! That's what my next post is about :)
- ashton314 2y ago
  I see from your bio that you are a PhD student. What are you doing with choreographies? (I’m in this space too.)
- lachlan_gray 2y ago
  This could be a fun example to work with :p https://en.m.wikipedia.org/wiki/Shakespeare_Programming_Language https://en.m.wikipedia.org/wiki/Shakespeare_Programming_Lang...
  - shadaj 2y ago
    You might enjoy my first ever blog post from ~10 years ago, when I first learned about distributed systems: https://www.shadaj.me/writing/romeo-juliet-and-reactive-programming https://www.shadaj.me/writing/romeo-juliet-and-reactive-prog...
roadbuster 2y ago
How does "choreographic programming" differ from the actor model?
- LegionMammal978 2y ago
  From what I can tell, the important distinction is that all actors (and their messages) are described alongside each other, instead of being described separately. There are many implementations of the actor model, but most of them are the 'static-location architectures' that TFA talks about.

cmrdporcupine 2y ago

Two things: Distributed systems are difficult to reason about. Computer hardware today is very powerful. There is a yo-yo process in our industry over the last 50 years between centralization and distribution. We necessarily distribute when we hit the limits of what centralization can accomplish because in general centralization is easier to reason about. When we hit those junctures, there's a flush of effort into distributed systems. The last major example of this I can think of was the 2000-2010 period, when MapReduce, "NoSQL" databases, Google's massive arrays of supposedly identical commodity grey boxes (not the case anymore), the High Scalability blog, etc. were the flavour of the time. But then, frankly, mass adoption of SSDs, much more powerful computers, etc. made a lot of those things less necessary. The stuff that most people are doing doesn't require a high level of distributed systems sophistication. Distributed systems are an interesting intellectual puzzle. But they should be a means to an end not an end in themselves.

porridgeraisin 2y ago
I think the reason that distributed systems still are the go-to choice for many software teams is to do with people/career expectations/careers orienting themselves around distributed systems over the time period you mentioned. It will take a while for it to re-orient, and then distributed systems might become a fad again ;) An example of this is typical promotion incentives being easier to get in microservice teams, thereby incentivising people to organize the team/architecture in that way.
- cmrdporcupine 2y ago
  Honestly, I am more cynical and just think people are always looking for ways to make their jobs more interesting than they actually are.
  - Nevermark 2y ago
    > Honestly, I am more cynical and just think people are always looking for ways to make their jobs more interesting than they actually are. When you frame the problem that way, unnecessary complexity seems like part of a healthy solution path. /h Companies get reliability benefits from slack, but creative people abhor wasted slack. Some basic business strategy/wisdom for maintaining/managing creative slack is needed.

taeric 2y ago

I'm not clear what the proposal here is? It specifically eschews tooling as a driver in the solution, but why? Wouldn't tooling be one of the most likely areas to get solid progress, as you could make tooling and point it at existing products. Would be interesting to see comparisons to other domains. Surely you could look at things like water processing plants to see how they build and maintain massive structures that do coordinated work between parts of it? Power generation plants. Assembly factories. Do we not have good artifacts for how these things are designed and reasoned about?

sriram_malhar 2y ago

The X10 language captures the notion of location of computation as a language primitive ("place") http://x10-lang.org http://x10-lang.org

tracnar 2y ago

The unison programming language does foray into a truly distributed programming language: https://www.unison-lang.org/ https://www.unison-lang.org/

aDyslecticCrow 2y ago
Functional programming languages already have alot of powerful concepts for distributed programming. Loads of the distributed programming techniques used elsewhere are often taken from an obscure fp language from years prior. Erlang comes to mind as still quite uniquely distributed without non fp comparison Unison seems to build on it further. Very cool

hinkley 2y ago

I don’t think there’s anyone in the Elixir community who wouldn’t love it if companies would figure out that everyone is writing software that contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang, and start hiring Elixir or Gleam devs. The future is here, but it is not evenly distributed.

deleted 2y ago
[deleted]
ikety 2y ago
It's so odd seeing people dissuade others for implementing "niche" languages like Elixir or Gleam. If you post a job opportunity with these languages, I guarantee you will be swamped with qualified candidates that are very passionate and excited to work with these languages full time.
- hinkley 2y ago
  At this point I’m worried that because elixir is over 10 years old that it’ll never arrive. But then Python is older than Java and here we are.
ignoramous 2y ago
> writing software that contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang Since you were at AWS (?), you'd know that Erlang did get its shot at distributed systems there. I'm unsure what went wrong, but if not c/c++, it was all JVM based languages soon after that.
- hinkley 2y ago
  No I worked a contract in the retail side and would not wish that job on anyone. My most recent favorite boss works there now and I haven’t even said hi because I’m afraid he’ll offer me a job.
- worthless-trash 2y ago

margorczynski 2y ago

Distributed systems are cool but most people don't really get how much complexity it introduces which leads them to fad-driven decisions like using Event Sourcing where there is no fundamental need to use it. I've seen projects getting burned because of the complexity and overhead it introduces where "simpler" approaches worked well and were easy to extend/fix. Hard to find and fix bugs, much slower feature addition and lots of more goodies the blogs with toy examples don't speak about.

nine_k 2y ago
The best recipe I know is to start from a modular monolith [1] and split it when and if you need to scale way past a few dozen nodes. Event sourcing is a logical structure; you can implement it with SQLite or even flat files, locally, if you your problem domain is served well by it. Adding Kafka as the first step is most likely a costly overkill. [1]: https://awesome-architecture.com/modular-monolith/ https://awesome-architecture.com/modular-monolith/
- margorczynski 2y ago
  What you're speaking of is a need/usability-based design and extension where you design the solution with certain "safety valves" that let you scale it up when needed. This is in contrast to the fad-driven design and over-engineering that I'm speaking of (here I simply used ES as an example) that is usually introduced because someone in power saw a blog post or 1h talk and it looked cool. And Kafka will be used because it is the most "scalable" and shiny solution, there is no pros-vs-cons analysis.
rjbwork 2y ago
If the choice has already been made to do a distributed system (outside of the engineer's control...), is a choice to use Event Sourcing by the engineer then a good idea?
mrkeen 2y ago
In my experience: 1) We are surrounded by distributed systems all the time. When we buy and sell B2B software, we don't know what's stored in our partners databases, they don't know what's in ours. Who should ask whom, and when? If the data sources disagree, whose is correct? Just being given access to a REST API and a couple of webhooks is all you need to be in full distributed systems land. 2) I honestly do not know of a better approach than event-sourcing (i.e. replicated state machine) to coordinate among multiple masters like this. The only technique I can think of that comes close is Paxos - which does not depend on events. But then the first thing I would do if I only had Paxos, would be to use it to bootstrap some kind of event system on top of it. Even the non-event-sourcing technologies like DBs use events (journals, write-ahead-logs, sstables, etc.) in their own implementation. (However that does not imply that you're getting events 'for free' by using these systems.) My co-workers do not put any alternatives forward. Reading a database, deciding what action to do, and then carrying out said action is basically the working definition of a race-condition. Bankers and accountants had this figured out thousands of years ago: a bank can't send a wagon across the country with queries like "How much money is in Joe's account?" wait a week for the reply, and then send a second wagon saying "Update Joe's account so it has $36.43 in it now". It's laughable. But now that we have 50-150ms latencies, we feel comfortable doing GETs and POSTs (with a million times more traffic) and somehow think we're not going to get our numbers wrong. Like, what's an alternative? I have a shiny billion-dollar fully-ACID SQL db with my customer accounts in them. And my SAAS partner bank also has that technology. Put forward literally any idea other than events that will let us coordinate their accounts such that they're not able to double-spend money, or are prevented from spending money if a node is down. I want an alternative to event sourcing.

mgraczyk 2y ago

The author is missing information about LLMs. In the "Obligatory LLM Section" he focuses on distributed systems that use LLMs. But almost all of the new innovation I'm familiar with in distributed systems is about training LLMs. I wouldn't say the programming techniques are "new" in the way this post is describing them, but the specifics are pretty different from building a database or data pipeline engine (less message oriented, more heavily pipelined, more low level programming, etc)

riku_iki 2y ago
> But almost all of the new innovation I'm familiar with in distributed systems is about training LLMs I think database space is still hot topic with many unsolved problems.

sanity 2y ago

The article makes great points about why distributed programming has stalled, but I think there's still room for innovation—especially in how we handle state consistency in decentralized systems. In Freenet[1], we’ve been exploring a novel approach to consistency that avoids the usual trade-offs between strong consistency and availability. Instead of treating state as a single evolving object, we model updates as summarizable deltas—each a commutative monoid—allowing peers to merge state independently in any order while achieving eventual consistency. This eliminates the need for heavyweight consensus protocols while still ensuring nodes converge on a consistent view of the data. More details here: https://freenet.org/news/summary-delta-sync/ https://freenet.org/news/summary-delta-sync/ Would love to hear thoughts from others working on similar problems! [1] https://freenet.org/ https://freenet.org/

Karrot_Kream 2y ago
Haven't read the post yet (I should, I have been vaguely following y'all along but obviously not close enough!) How is this different from delta-based CRDTs? I've built (admittedly toy) CRDTs as DAGs that ship deltas using lattice operations and it's really not that hard to have it work. There's already CRDT based distributed stores out there. How is this any different?
- sanity 2y ago
  Good question! Freenet is a decentralized key-value store, but unlike traditional KV stores, the keys are WebAssembly (WASM) contracts. These contracts define not just what values (i.e., data or state) are valid for that key but also when and how the value can be mutated. They also specify how to efficiently synchronize the value across peers using summaries and deltas. Each contract determines how state changes are validated, summarized, and merged, meaning you can efficiently implement almost any CRDT mechanism in WASM on top of Freenet. Another key difference is that Freenet is an observable KV store, allowing you to subscribe to values and receive immediate updates when they change.
  - Karrot_Kream 2y ago

MisterTea 2y ago

This has been an issue for quite some time. Rob Pike wrote about it going back - https://doc.cat-v.org/bell_labs/utah2000/ https://doc.cat-v.org/bell_labs/utah2000/

bigmutant 2y ago

Good resources for understanding Distributed Systems: - MIT course with Robert Morris (of Morris Worm fame): https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39... - Martin Kleppmann (author of DDIA): https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjc... If you can work through the above (and DDIA), you'll have a solid understanding of the issues in Distributed System, like Consensus, Causality, Split Brain, etc. You'll also gain a critical eye of Cloud Services and be able to articulate their drawbacks (ex: did you know that replication to DynamoDB Secondary Indexes is eventually consistent? What effects can that have on your applications?)

ignoramous 2y ago
> Robert Morris (of Morris Worm fame) (of Y Combinator fame, too)

Karrot_Kream 2y ago

When I was graduating from my Masters (failed PhD :) this overview of various programming models is generally how I thought of things. I've been writing distributed code now in industry for a long time and in practice, having worked at a some pretty high-scale tech companies over the years, most shops tend to favor static-location style models. As the post states, it's due largely to control and performance. Scaling external-distribution systems has been difficult everywhere I've seen it tried and usually ends up creating a few knowledgeable owners of a system with high bus-factor. Scaling tends to work fine until it doesn't and these discontinuous, sharp edges are very very painful as they're hard to predict and allocate resourcing for. Are external-distribution systems dead ends then? Even if they can achieve high theoretical performance, operation of these systems tends to be very difficult. Another problem I find with external-distribution systems is that there's a lot of hidden complexity in just connecting, reading, and writing to them. So you want to talk to a distributed relational DB, okay, but are you using a threaded concurrency model or an async concurrency model? You probably want a connection pool so that TCP HOL blocking doesn't tank your throughput. But if you're using threads, how do you map your threads to the connections in the pool? The pool itself represents a bottleneck as well. How do you monitor the status of this pool? Tools like Istio strive to standardize this a little bit but fundamentally we're working with 3 domains here just to write to the external-distribution system itself: the runtime/language's concurrency model, the underlying RPC stack, and the ingress point for the external-distribution system. Does anyone have strong stories of scaling an external-distribution system that worked well? I'd be very curious. I agree that progress here has stalled significantly. But I find myself designing big distributed architecture after big distributed architecture continuing to use my deep experience of architecting these systems to build static-location systems because if I'm already dealing with scaling pains and cross-domain concerns, I may as well rip off the band-aid and be explicit about crossing execution domains.

KaiserPro 2y ago

Distributed systems are hard, as well all know. However the number of people that actually need a distributed system is pretty small. With the rise of kubernetes, the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped. You go distributed either because you are desperate, or because you think it would be fun. K8s takes the fun out of most things. Moreover, with machines suddenly getting vast IO improvements, the need for going distributed is much less than it was 10 years. (yes i know there is fault tolerance, but that adds another dimension of pain.)

sd9 2y ago
> the number of people who've not been burnt by going distributed when they didn't need to has rapidly dropped Gosh, this was hard to parse! I’m still not sure I’ve got it. Do you mean “kubernetes has caused more people to suffer due to going distributed unnecessarily”, or something else?
- boarush 2y ago
  Had me confused for a second too, but I think it is the former that they meant. K8s has unneeded complexity which is really not required at even decent enough scales, if you've put in enough effort to architect a solution that makes the right calls for your business.
  - KaiserPro 2y ago
    yeah sorry, double negatives. People got burnt by kubernetes, and that pissed in the well of enthusiasm for experimenting with distributed systems
    - DrFalkyn 2y ago
      Because people, especially Devops, thought k8s was some magic, when all it really does is makes the mechanics easier If you’re architecture is poor k8s won’t help you
2y ago

synergy20 2y ago

I saw comments about embedded development, which I have been doing that for a long time, just want to make a point here: the pay has upper limits, you will be paid fine but will reach the pay limit very fast, and it will stay there for the rest of your career. they can swap someone in with that price tag to do whatever you are working on, because, after all, embedded devel is not rocket science.

cmrdporcupine 2y ago
The problem with embedded is its proximity to EE which is frankly underpaid. But it's also more that the "other" kind of SWE work -- "backend" etc is frankly overpaid because of the copious quantities of $$ dumped into it by VC and ad money.

sn9 2y ago

It sounds like they're looking for the Rust project Hydro [0]. EDIT: Lol nvm the author is one of the authors of Hydro [1]. [0] https://github.com/hydro-project/hydro https://github.com/hydro-project/hydro [1] https://hydro.run/people https://hydro.run/people

seivan 2y ago
[dead]

jderick 2y ago

Distributed systems are hard. I like the idea of "semantic locality." I think it can be achieved to some degree via abstraction. The code that runs across many machines does a lot of stuff but only a small fraction of that is actually involved in coordination. If you can abstract away those details you should end up with a much simpler protocol that can be modeled in a succinct way. Then you can verify your protocol much more easily. Formal methods have used tools such as spin (promela) or guarded commands (murphi) for modeling these kinds of systems. I'm sure you could do something similar with the lean theorem prover. The tricky part is mapping back and forth between your abstract system and the real one. Perhaps LLMs could help here. I work on hardware and concurrency is a constant problem even at that low level. We use model checking tools which can help.

lifeisstillgood 2y ago

This is a massive coming issue - I am not sure “distributed” can be exactly replaced with “parallel processing” but it’s close So to simplify, from 1985 to 2005 ish you could keep sequential software exactly the same and it just ran faster each new hardware generation. One CPU but transistors got smaller and (hand wavy, on chip ram, pipelining ) Then roughly around 2010 single CPUs just stopped magically doubling. You got more cores, but that meant parallel or distributed programming - your software that in 1995 served 100 people was the same serving 10,000 people in 2000. But in 2015 we needed new coding - we got NOSQL and map reduce and facebook data centres. But the hardware kept growing TSMC now has wafer scale chips with 900,000 cores - but my non parallel, on distributed code won’t run 1 million times faster - Amdahls law just won’t let me So yeah - no one wants to buy new chips with a million cores because you aren’t going to get the speed ups - why buy an expensive data centre full of 100x cores if you can’t sell them at 100x usage.

kodablah 2y ago

> Just like the external-distribution model, arbitrary-location architectures often come with a performance cost. Durable execution systems typically snapshot their state to a persistent store between every step. This is not true by most definitions of "snapshot". Most (all?) durable execution systems use event sourcing and therefore it's effectively an immutable event log. And it's only events that have external side effects enough to rebuild the state, not all state. While technically this is not free, it's much more optimal than the traditional definition of capturing and storing a "snapshot". > But this simplicity comes at a significant cost: control. By letting the runtime decide how the code is distributed [...] we don’t want to give up: Explicit control over placement of logic on machines, with the ability to perform local, atomic computations Not all durable execution systems require you to give this up completely. Temporal (disclaimer: my employer) allows grouping of logical work by task queue which many users use to pick locations of work, even so far as a task queue per physical resource which is very common for those wanting that explicit control. Also there are primitives for executing short, local operations within workflows assuming that's what is meant there.

anonymousDan 2y ago

This article is just word salad. In what way does Redis 'abstract distribution' for example?

gregw2 2y ago

What I noticed missing in this analysis of distributed systems programming was a recognition/discussion of how distributed databases (or datalakes) decoupling storage from compute have changed the art of the possible. In the old days of databases, if you put all your data in one place, you could scale up (SMP) but scaling out (MPP) really was challenging. Nowdays, you (iceberg), or a DB vendor (Snowflake, Databricks, BigQuery, even BigTable, etc), put all your data on S3/GCS/ADLS and you can scale out compute to read traffic as much as you want (as long as you accept something like a snapshot isolation read level and traffic is largely read-only or writes are distributed across your tables and not all to one big table.) You can now share data across your different compute nodes or applications/systems by managing permissions pointers managed via a cloud metadata/catalog service. You can get microservice databases without each having completely separate datastores in a way.

thway15269037 2y ago

Oh god, even this article has AI and LLM section in it. When I thought distributed system design could not get any worse, someone actually pitched AI slop in it. God I want to dig a cave and live in it.

Nevermark 2y ago
> When I thought distributed system design could not get any worse, someone actually pitched AI slop in it. I am not sure that pointing out that today's models are going to be MUCH worse at reasoning about distributed code than serial code is "pitching". Conversely, pointing out that the reason they are so bad at distributed is the lack of related information locality, the same problem humans often have, puts a reasonable second underline on the value of more locality in our development artifacts.

hintymad 2y ago

This reminds me of Rob Pike's article "Systems Software Research is Irrelevant," written about 15 years ago. Perhaps many systems have matured to a point where any improvement appears incremental to engineers, so the conviction to develop a new programming model isn't strong enough. Or perhaps we're in a temporary plateau, and a groundbreaking tool will emerge in a few years. Regarding Laddad's point, building tools native to distributed systems programming might be intrinsically difficult. It's not for lack of trying. We've invented numerous algebras, calculi, programming models, and experimental programming languages over the past decades, yet somehow none has really taken off. If anything, I'd venture to assert that object storage, perhaps including Amazon DynamoDB, has changed the landscape of programming distributed systems. These two systems, which optimize for throughput and reliability, make programming distributed systems much easier. Want a queue system? Build on top of S3. Want a database? Focus on query engines and outsource storage to S3. Want a task queue? Just poll DDB tables. Want to exchange states en masse? Use S3. The list goes on. Internally to S3, I think the biggest achievement is that S3 can use scalability to its advantage. Adding a new machine makes S3 cheaper, faster, and more reliable. Unfortunately, this involves multiple moving parts and is therefore difficult to abstract into a tool. Perhaps an arbitrarily scalable metadata service is what everyone could benefit from? Case in point, Meta's warm storage can scale to multiple exabytes with a flat namespace. Reading the paper, I realized that many designs in the warm storage are standard, and the real magic lies in its metadata management, which happens to be outsourced to Meta's ZippyDB. Meanwhile, open-source solutions often boast about their scalability, but in reality, all known ones have certain limits, usually no more than 100PBs or a few thousand nodes.

jbarham 2y ago
> This reminds me of Rob Pike's article "Systems Software Research is Irrelevant," written about 15 years ago. 25 years ago: http://herpolhode.com/rob/utah2000.pdf http://herpolhode.com/rob/utah2000.pdf (Time flies.)
- hintymad 2y ago
  Wow, time flies. Thanks!

nyrikki 2y ago

> Distributed SQL Engines This is what I see holding some applications back. The relational model is flexible and sufficient for many needs but the ACID model is responsible for much of the complexity in some more recent solutions. While only usable for one-to-many relationships, the hierarchical model would significantly help in some of the common areas like financial transactions. Think IBM IMS fastpath, and the related channel model. But it seems every neo paradime either initially hampers itself, or grows to be constrained by Codd's normalization rules, which result in transitive closure a the cost of independence. As we have examples like Ceph's radios, Kafka etc...if you view the hierarchical file path model as being intrinsic to that parent child relationship we could be distributed. Perhaps materialized views could be leveraged to allow for SQL queries without turning the fast path into a distributed monolith. SQL is a multi tool, and sometimes you just need to use a specific tool.

th0ma5 2y ago

Since multicore processing a ton of software you use or create is distributed you have to ask if you want to be in control of how it is distributed or not. If you want it easy and let the library figure it out then you have to accept the topological ideas it has. For instance H2O is a great machine learning package that even has its own transparent multi core processing. If you want to go across machines it has its own cluster built in. You can also install it into Hadoop, Spark, etc but once you start going that direction you're more and more on the hook for what that means and if it even is more effective for your problem and what your distributed strategy should be. Things like re-entrant idempotence, software transactional memory, copy on write, CRDTs etc are going to have waste and overhead but can vastly simplify conceptually the ongoing development and maintenance of even non-distributed efforts in my opinion, and we keep having the room to eat the overhead. There's a ton of bias against this for good reasons that the non distributed concepts still just work without any hassle but we'd be less in the mud in a fundamental way of we learned to let go of non-eventual consistency.

samonurofil 2y ago

[dead]

rstuart4133 2y ago

> Although static-location architectures offer developers the most low-level control over their system, in practice they are difficult to implement robustly without distributed systems expertise. This is the understatement of the article. There are two insanely difficult things to get right in computers. One is cryptography, and other is distributed systems. I'd argue the latter is harder. The reason simple enough to understand. In any program the programmer has to carry in his head every piece of state that is accessible at any given point, the invariants that apply to that state, and the code responsible for modifying that state while preserving the invariants. In sequential programs the code that can modify the shared state is restricted to inner loops and functions you call, and you have to verify every modification preserves the invariants. It's a lot. The hidden enemy is aliasing, and you'll find entire books written on the counter measures like immutable objects, function programming, and locks. Coordinating all this is so hard only a small percentage of the population can program large systems. I guess you are thinking "but of a lot of people here can do that". True, but we are a tiny percentage. In distributed systems those blessed restrictions a single execution thread gives us on what code can access shared state goes out the window. Every line that could read or write the shared state has to be considered, whether its adjacent or not, whether you called it here or not. The state interactions explode in the same way interactions between qubits explode. Both explode beyond the capability of human minds to assemble them all in one place. You have to start forming theorems and formulating proofs. That worst part is newbie programmers are not usually aware this explosion has taken place. That's why experienced software engineers give the following advice on threads: just don't. You don't have a feel for what will happen, your code will appear to work when you test it while being rabbit warren of disastrous bugs that will likely never be fixed. It's why Linux RCU author Paul McKenney is still not confident his code is correct, despite being one of the greatest concurrent programming minds on the planet. It's why Paxos is hard to understand despite being relatively simple. Expecting an above average programmer to work on a distributed system and not introduce bugs without leaning on one of one of the "but it is inefficient" tools he lists is an impossible dream. A merely experienced average has no hope. It's hard. Only a tiny, tiny fraction of the programmers on the planet can pull it off kind of hard.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

sakesun 2y ago

I have been told and believed that we are getting very close to the ultimate answer throughout my career since CORBA/DCOM. Just learn that there is another discontinued attempt https://serviceweaver.dev/ https://serviceweaver.dev/

nchammas 2y ago

There is an old project out of Berkeley called BOOM [1] that developed a language for distributed programming called Bloom [2]. I don't know enough about it to map it to the author's distributed programming paradigms, but the Bloom features page [3] is interesting: > disorderly programming: Traditional languages like Java and C are based on the von Neumann model, where a program counter steps through individual instructions in order. Distributed systems don’t work like that. Much of the pain in traditional distributed programming comes from this mismatch: programmers are expected to bridge from an ordered programming model into a disordered reality that executes their code. Bloom was designed to match–and exploit–the disorderly reality of distributed systems. Bloom programmers write programs made up of unordered collections of statements, and are given constructs to impose order when needed. [1]: https://boom.cs.berkeley.edu https://boom.cs.berkeley.edu [2]: http://bloom-lang.net/index.html http://bloom-lang.net/index.html [3]: http://bloom-lang.net/features/ http://bloom-lang.net/features/

jmhucb 2y ago
Good pattern matching. Bloom is a predecessor project to the OP's PhD thesis work :-) This area takes time and many good ideas to mature, but as the post hints, progress is being made.

ConanRus 2y ago

No Erlang mention? Sad.

anacrolix 2y ago

I've been trying to explain this to people for 8 years. All of our existing languages side step the problem. Developers are writing distributed systems every day but seem oblivious to the fact their tools aren't helping at all.

hackburg 2y ago

[dead]

lincpa 2y ago

[dead]