21 ms·

Fragile narrow laggy asynchronous mismatched pipes kill productivity

264 points by trishume 6y ago

smitty1e 6y ago
> Sometimes a distributed system is unavoidable, such as if you want extreme availability or computing power, but other times it’s totally avoidable. But so much of our sales pitch involves these shiny cloud systems. Who ever sold business by telling the customer: "Your use-case really isn't exciting, and a boring batch-driven process is completely appropriate"?
- TomMarius 6y ago
  I do it like that, it's always met with excitement like "oh wow, all these other companies were telling us how hard and costly and lengthy it will be, thank you"
  - smitty1e 6y ago
    Do you ever have the experience of doing a prototype, and the customer looks at it and says: "Great work. Put it into production"?
    - dylan604 6y ago
      every. damn. time. i'm a freelancer, so there's not a dev team behind me. the client likes what i made, and wants to start using it. realizing this, i've started spending much more time on the UI/UX (even though i'm not one of those guys) to at least make the tool useable in a dogfooding way.
      - MauranKilom 6y ago
        Maybe it's not 100% relevant to your case, and maybe you've already seen it, but just in case: https://www.joelonsoftware.com/2002/02/13/the-iceberg-secret-revealed/ https://www.joelonsoftware.com/2002/02/13/the-iceberg-secret...
        john-tells-all 6y ago
        This article is spectacular, thank you! Joel explains that it's hard explaining Dev work to nondevs, e.g. the Business. They think the UI/screenshot is the software, which has a number of consequences. As a DevOps, I also see a similar challenge explaining Infra/DevOps to Developers :)
        ChrisMarshallNY 6y ago
        Joel Spolsky is my hero. There are so many nails he’s been hitting on the head for decades. Engineering (to me) is a craft; not just a vocation. I also think that the software industry’s Ponce de Leon obsession might be part of the problem. Younger folks tend to “dream big,” which I think may be a factor behind the ageism. “They haven’t had some grizzled old fart tell them that it can’t be done!” is the argument that I’ve heard. Tru dat, but one of the things that older folks have, is experience getting things done, often, highly optimized, polished, and of excellent quality. That tends to require a lot of tenacity and patience. Shipping is boring. Ever watch a building go up? A good prefab looks complete after three months, but doesn’t open for another nine months. It looks awesome and shiny, but is still behind a rent-a-fence. What gives? That’s because all that interior work; the finish carpentry, the drywall, the painting, etc., take forever, and these are the parts of the building that see everyday use, so they need to be absolutely spot-on. The outside is mostly a pigeon toilet. It doesn’t need to be as complete; a solid frame and watertight is sufficient. They just needed it to keep the rain out, while the really skilled craftspeople got their jobs done. I like to make stuff polished, tested and complete. I don’t like making pigeon toilets.
- FpUser 6y ago
  >Who ever sold business by telling the customer: "Your use-case really isn't exciting, and a boring batch-driven process is completely appropriate"? I did many times. Many practical businesses are looking how to solve their real problems in reliable and cheapest way. They mostly do not give a flying hoot about all those buzzwords and super duper newest tech. They just ask how much, how exactly will it work, what are hidden cost, how will it be maintained and they will also look at your profile (clients, references, examples of finished projects etc).
lostcolony 6y ago
Failure is such a fun thing to think about, and it gets handwaved away so often. So many devs, architects, product owners, etc, just focus on happy path, and leave failure unspecced, unhandled, and just hope it never happens. And then boast about 99% uptime, but once you start questioning them you find out they get weekly pages they have to go investigate (and really the system is behaving weirdly a solid 10% of the time, but they don't know what to do about it and it eventually resolves itself, and they don't count "pageable weirdness" in their failure metric). It's actually one of the things I love about Erlang, and how it's changed my thinking. Think about failures. Or rather; don't. Assume they'll happen, in ways you can't plan for. Instead think about what acceptable degraded behavior looks like, how to best ensure it in the event of failure, and how to automatically recover.
- deleted 6y ago
  [deleted]
- LargeWu 6y ago
  As a developer who's trying to move on to either a management or product role, failure modes is one of the things I want to emphasize in that role. The sad fact is that so many product owners really don't understand how software works or gets built, and as such, they are unequipped to reason about such things.
- jodrellblank 6y ago
  On the subject of failures, you might like this blog post https://danluu.com/postmortem-lessons/ https://danluu.com/postmortem-lessons/ if you haven't seen it before.
adrianmonk 6y ago
See also Peter Deutsch's "Fallacies of Distributed Computing" list (https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...). There's some overlap, but also some new stuff. In particular, "pipes" isn't covered by the Fallacies list and is consistently a pain point and/or issue you always face in some way. Also "asynchronous" isn't covered by the Fallacies list.
wpietri 6y ago
Funnily, I thought the headline was talking about development process, as that also describes how a lot of places (mis-)handle the flow of what get worked on.
deleted 6y ago
[deleted]
evadne 6y ago
Do you have a moment to talk about our Lord and Saviour, Erlang/OTP?
- shijie 6y ago
  My thoughts exactly! As the Elixir code at my company continues to grow at a rapid rate, writing OTP services for everything has allowed me to never have to think about entire classes of bugs and edge cases simply by virtue of the patterns inherent in OTP and Elixir/Erlang. Right tool for the right job.
chubot 6y ago
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That is, most of a whole Linux distro. I have been loosely following some "git for binary data" projects for a number of years. I looked at IPFS like 5 years ago but it seems to have gone off the rails. The dat project seems to have morphed into something else? Are there any new storage projects in that vein? I think the OP is identifying a real problem -- distributed systems are unreliable, and you can get a lot done on a single machine. But we are missing some primitives that would enable that. Every application is littered with buggy and difficut network logic, rather than having a single tool like git (or rsync) which would handle the problem in a focused and fast way. It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue. Sort of related is a fast distro here I'm looking at: https://michael.stapelberg.ch/posts/2019-08-17-introducing-distri/ https://michael.stapelberg.ch/posts/2019-08-17-introducing-d...
- trishume 6y ago
  This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
  - chubot 6y ago
    Yeah I agree, it's sort of an open problem, but I guess a bunch of arrows are pointing toward FUSE. Distri uses FUSE and it appears Microsoft's GVFS uses FUSE or whatever Windows technology is the equivalent. (My teammates developed Google's equivalent about 14 years ago, using FUSE, so it's something I've used / seen several times.) FUSE requires some kernel support (a module), which git of course doesn't require. That is a barrier, but perhaps not an insurmountable one. Basically I would like to offload all the "network" work to the OS, so applications are free of that logic.
perfunctory 6y ago
Just reading the title I assumed it was a post about business processes and communication between teams. Because this is how working for a big corp sometimes feel.
- snazz 6y ago
  It's a bit of Conway's law: the software structure mirrors the organization structure.
tlarkworthy 6y ago
I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It's how you outsource expertise. The vision of the service mesh is to make unreliability and security no longer the job if the application binary. Even without a service mesh, you can put a lot of common functionality into a reverse proxy. Personally I am loving OpenResty for the simplicity of writing adapters and oauth at the proxy layer with good performance.
- trishume 6y ago
  I think it should be possible to buy or use software from third parties. One thing I'm disappointed about and think we need better tools to avoid is the fact that third parties provide their tools as services rather than libraries. There's reasons they do that, deploying a library that all your customers can easily use is hard right now, but there's no reason it has to be that way. Some things can't easily be libraries like databases, but other things like some caches, miscellaneous operations like image resizing (depending on an architecture that can handle the load on those servers), and a bunch of other things could just be libraries.
  - nemetroid 6y ago
    > Some things can't easily be libraries like databases The world's most widely deployed database engine (SQLite) is only available as a library.
    - tlarkworthy 6y ago
      The worlds most trafficked database is probably Google search* index and its not sqlite and it's certainly distributed. * Maybe Facebook, I dunno, the point stands.
      - 6y ago
gfxgirl 6y ago
doh! I thought this was going to be about remote work.
at_a_remove 6y ago
Yes, I have a little Python library for managing network shares in Windows. It has things like automatic retries that "back off" slowly, switching to cached IPs in case DNS is down, and checking to see if all of the drive-letters are full and either re-using a letter or creating a "letter-less" share. I had to develop it during a period of great instability within our network. It's ... large and over-engineered, but it just keeps on truckin'. On the other hand, it has been quite useful going forward, so that's a plus. I tend to program fairly defensively, in layers, right down to the much maligned Pokemon exception handling. The results don't have the, ah, velocity that is so often praised but they'll be there ticking along years later.
crazygringo 6y ago
Of course they do. But there's no alternative. No matter how fast or beefy your server is, these days if your product becomes a success, 99% of the time it will outgrow what's possible on a single server. (Not to mention needs for redundancy, geographic latency, etc.) And by the time you see the trend heading upwards so you can predict what day that will happen, you already won't have the time for the massive rewrite necessary. So yes, it's tons slower to write distributed servers/systems. But what other choice do you usually have? Though, as much as possible, you can try to avoid the microservices route, and integrate everything as much as possible into monolithic replicable "full-stack servers" that never talk to each other, but rather rely entirely on things like cloud storage and cloud database. Where you're paying your cloud provider $$$ to not fail, rather than handle it yourself. Sometimes this will work for you, sometimes it won't.
- nixpulvis 6y ago
  I've seen a handful of applications that attempt to "scale" by going down the micro-service route in a completely flawed way. Only to end up with a tangled mess that's impossible to reliably debug. All progress halts. There's nothing inherently wrong about your statement, just that it's still far to easy to write shitty distributed system, and so much easier to push that complexity off onto the OS or even network layer itself. Why should I the application developer care about the way my user's data enters my DB? This should be tightly abstracted away, and traced/logged accordingly. Leave it to me the systems developer to get the details right, and share the fruits of my labor with everyone. I can imagine no system more deserving of shared resources than network technology. Try and imagine a world without TCP/IP, do you not end up with something similar?
- lowbloodsugar 6y ago
  I think there are plenty of examples of teams that hit that cliff and had to rewrite. I am trying to remember specific articles. Counter-argument: I think stackoverflow hit a cliff, and remained monolithic, but that was several years ago [1] (sure they have a ton of servers, but they appear to be classic cache+web-server+database). There was another, maybe Instagram, that was monolithic and had to figure out how to shard their DB overnight. [1] https://nickcraver.com/blog/2016/03/29/stack-overflow-the-hardware-2016-edition/ https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha... Also, I don't think your post deserved a down-vote.
carapace 6y ago
> I hope this leads you to think about the ways that your work could be more productive if you had better tools to deal with distributed systems, and what those might be. We have tools. Promula/SPIN model checker is one just off the top of my head.
btreecat 6y ago
So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses. Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago. Then there are some folks/organizations building cathedrals, and the associated tooling (react, angular, maven, etc) and all the rest of us look up in awe and thing "well I guess if I want to be that good I need to use those tools on this dog house." But your dog house doesn't have the need to host parties, provide security, or even real weather protection other than a roof to keep the rain and sun out. Yet we all try to build our dog houses in ways that might be better if they are one day converted to a proper living quarters but likely will never have a need for running Water or windows.
- colechristensen 6y ago
  French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined with how the shell worked being driven by people who really really understood language and came up with one that made sense to them for interacting with computers... was, is still somewhat, wonderful. What's missing today is exactly what you mention. People designing tool sheds with materials often shoddily designed for cathedrals. Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. The favourite example of this is - perhaps apocryphal, but entirely believable - is replacing dozens of nodes using fancy big data tools with one node running sed/awk,etc. One thing is clear, nowhere I've been has had the tools readily available, the documentation clear and forthcoming, and the scale in the right range for projects. I found myself recently solving a problem with Hashicorp Vault using GCP to verify identity of machines wanting secrets. It was a stretch goal which had been on my plate for six months, every once in a while I would try to go back and figure out how to make it work, and months and months and months after trying, I put it together and it worked perfectly. The documentation to lead to this understanding had to be read out of order on several different pages with some lucky guesses to arrive at the solution, which in the end was just a few steps easily explained. Afterwards the documentation was fine and made perfect sense. Before I grokked the issue the documentation just seemed like a bunch on nonsense which led me to believe what I wanted to do wasn't possible in a constrained security environment. That is the kind of problem I solve all the time as someone with a decade of DevOps,SysAdmin,whatever experience behind me. Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating.
hn_throwaway_99 6y ago
Hah, I saw the title "Fragile narrow laggy asynchronous mismatched pipes kill productivity" and thought it was about the pitfalls of trying to coordinate remote teams across disparate time zones.
- acjohnson55 6y ago
  Yeah, my guess was it was a Slack takedown :)
ChrisMarshallNY 6y ago
This article has a point. But, as in all things software, "it depends." It depends on what the tools are, and what we are writing. In my own case, I have the luxury of writing fully native Swift code for Apple devices. I don't need to work with anything off the device, except for fairly direct interfaces, like USB, TCP/IP or Bluetooth. Usually. I have written "full stack" systems that included self-authored SDKs in Swift, and self-authored servers in PHP/JS. I avoid dependencies like the plague. Some of them are excellent, and well worth the effort, but I have encountered very few that really make my life as an Apple device programmer that much easier. The rare ones I do use (like SOAPEngine, or ffmpeg, for instance), are local to the development environment, and usually quite well-written and supported. If I were writing an app that reflected server-provided utility on a local device, then there's a really good chance that I'd use an SDK/dependency with network connectivity, like GraphQL, or MapBox. These are great services, but ones that I don't use (at the moment). I'm skeptical of a lot of "Big Social Media" SDKs. I believe that we just had an issue with the FB SDK. That said, if I were writing an app that leveraged FB services, I don't see how I could avoid their SDK. So I write fully native software with Swift, and avoid dependencies. That seems to make my life easier. But Xcode is still a really crashy toolbox.
nitwit005 6y ago
I've found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades or maintenance. Common example is data being pushed between systems with HTTP. Take the simplest case of propagating a boolean value. You toggle some setting in the UI, and it sends an update to another system with an HTTP request, retrying on a delay if it can't connect. This has two problems. The first is that if the user toggles a setting on and then off, you can have two sets of retries going, producing a random result when the far end can be connected to again. The second, is that the machine doing the retries might get rebooted, and people often fail to persist the fact that a change needs to be pushed to the other system. I've seen this issue between two processes on the same machine, so technically you don't even need a network.
- jxcl 6y ago
  If you're dealing with multiple processes persisting data to different data stores, you're dealing with a distributed database problem, which has many more failure modes than the ones you've described here. If there's anything I've learned across my career, it's to avoid distributed databases unless absolutely necessary, and if it is necessary, then spend a bunch of time trying to make sure you got it right. And then even after that you probably got it wrong.
robbrown451 6y ago
The title reminded me of the turboencabulator. https://www.thechiefstoryteller.com/2014/07/16/turbo-encabulator-best-worst-jargon/ https://www.thechiefstoryteller.com/2014/07/16/turbo-encabul...
jancsika 6y ago
> Untrusted: If you don’t want everything to be taken down by one malfunction you need to defend against invalid inputs and being overwhelmed. Sometimes you also need to defend against actual attackers. Sorry, but unless your centralized alternative is only used internally by troglodytes you have to at least defend against invalid inputs.
Andaith 6y ago
Just some fun with the English language: > probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed. If you don't ever plan on distributing your software you can save a _lot_ of time :)
emmelaich 6y ago
It's perfectly applicable to people too; one stickler for the rules or slow working in a critical role, say security officer, or change board chair can kill productivity.
ahh 6y ago
I feel so attacked right now.
lowbloodsugar 6y ago
Really like the curiosity and thought behind this article. Couple of thoughts: >probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed. Sure. Do it on one giant machine. Then you'll be spending 80% of your time doing things you wouldn't need to do if it weren't monolithic. At the end of the day, if your customer is on the other end of the internet, then all of those complaints apply. If you solve that by running an app on their device, then oh boy are you going to have fun testing. I prefer scaling out. The stackoverflow peeps prefer scaling up. There are some great write-ups about how they scale. I found this [1] one after some quick googling, but I am certain there are more. So it's really about choose your poison. >I think people should be more willing to try and write performance-sensitive code as a (potentially multi-threaded) process on one machine in a fast language if it’ll fit rather than try and distribute a slower implementation over multiple machines. Sure. I once replaced a system that ran on 10 32-core machines with one that ran on one with four cores on one machine and did the work in the same time. Another time I had 96 cores, more threads, and I replaced it with one that had three threads and was faster. But both of those solutions were evolutionary dead-ends. The tasks were very specific, and not subject to change. The first one was a single C file. The latter was actually java, but with hand-rolled hash tables and optimistic locks. The first one I doubt I could follow it now. My point is, you can have understandable systems that good people (as opposed to geniuses) can work on, evolve and adapt, and that have well understood failure modes and scaling cliffs. Or you can have bonkers code that everyone is afraid to touch, and which fails in production when it hit a cliff you didn't know about and now your site is dead for eight days. If you can strike a good balance, then you'll probably have some combination of distributed, and brute force. [1] http://highscalability.com/stack-overflow-architecture http://highscalability.com/stack-overflow-architecture
- naniwaduni 6y ago
  > Sure. Do it on one giant machine. Then you'll be spending 80% of your time doing things you wouldn't need to do if it weren't monolithic. Deeply false equivalence.
_bxg1 6y ago
> I also think all these costs mean you should try really hard to avoid making your system distributed if you don’t have to. There's a point here about microservices.
lazyjones 6y ago
Networking and distributed software are well understood nowadays, the lower productivity of web companies vs. SpaceX etc. comes from all the unstable (both ever-changing and buggy) software they need to use. Most modern software is affected by this, but the web has it worse because of security issues and because the way browsers are evolving (on purpose, one has to add, because it's a cartel of large players on the web trying to stifle competition). SpaceX doesn't get some innovative new alloy they didn't order every couple of weeks and even the games industry has fewer obstructing external dependencies (hardware vendors and their drivers being one). At least that's my experience from about 20 years of web development (15 professionally).
steve76 6y ago
What's the opposite? "Sturdy broad quick harmonious complimentary pipes boost productivity" Details are inescapable, whether you are painting or composing or writing software. Ideas have holes in them and designs fall apart in your hands during implementation. The best approach I found is to just write down mistakes. I'm building software that reverses the freelance market. Instead of an endless list of buyers requests, there are dedicated stages buyers and programmers select, and everyone works together. Instead of tracking project goals, mistakes are track and there's specialized production stages to software design.
RandyRanderson 6y ago
TLDR he means microservices. We now have a generation that doesn't even recall a non-MS world. Tristan, the generation between us has created the IT world you describe. You'll probably spend the next 20 years of your career dealing with that mess. Sorry about that.

264 points by trishume 6y ago

> Sometimes a distributed system is unavoidable, such as if you want extreme availability or computing power, but other times it’s totally avoidable. But so much of our sales pitch involves these shiny cloud systems. Who ever sold business by telling the customer: "Your use-case really isn't exciting, and a boring batch-driven process is completely appropriate"?

TomMarius 6y ago
I do it like that, it's always met with excitement like "oh wow, all these other companies were telling us how hard and costly and lengthy it will be, thank you"
- smitty1e 6y ago
  Do you ever have the experience of doing a prototype, and the customer looks at it and says: "Great work. Put it into production"?
  - dylan604 6y ago
    every. damn. time. i'm a freelancer, so there's not a dev team behind me. the client likes what i made, and wants to start using it. realizing this, i've started spending much more time on the UI/UX (even though i'm not one of those guys) to at least make the tool useable in a dogfooding way.
    - MauranKilom 6y ago
      Maybe it's not 100% relevant to your case, and maybe you've already seen it, but just in case: https://www.joelonsoftware.com/2002/02/13/the-iceberg-secret-revealed/ https://www.joelonsoftware.com/2002/02/13/the-iceberg-secret...
      - john-tells-all 6y ago
        This article is spectacular, thank you! Joel explains that it's hard explaining Dev work to nondevs, e.g. the Business. They think the UI/screenshot is the software, which has a number of consequences. As a DevOps, I also see a similar challenge explaining Infra/DevOps to Developers :)
      - ChrisMarshallNY 6y ago
        Joel Spolsky is my hero. There are so many nails he’s been hitting on the head for decades. Engineering (to me) is a craft; not just a vocation. I also think that the software industry’s Ponce de Leon obsession might be part of the problem. Younger folks tend to “dream big,” which I think may be a factor behind the ageism. “They haven’t had some grizzled old fart tell them that it can’t be done!” is the argument that I’ve heard. Tru dat, but one of the things that older folks have, is experience getting things done, often, highly optimized, polished, and of excellent quality. That tends to require a lot of tenacity and patience. Shipping is boring. Ever watch a building go up? A good prefab looks complete after three months, but doesn’t open for another nine months. It looks awesome and shiny, but is still behind a rent-a-fence. What gives? That’s because all that interior work; the finish carpentry, the drywall, the painting, etc., take forever, and these are the parts of the building that see everyday use, so they need to be absolutely spot-on. The outside is mostly a pigeon toilet. It doesn’t need to be as complete; a solid frame and watertight is sufficient. They just needed it to keep the rain out, while the really skilled craftspeople got their jobs done. I like to make stuff polished, tested and complete. I don’t like making pigeon toilets.
FpUser 6y ago
>Who ever sold business by telling the customer: "Your use-case really isn't exciting, and a boring batch-driven process is completely appropriate"? I did many times. Many practical businesses are looking how to solve their real problems in reliable and cheapest way. They mostly do not give a flying hoot about all those buzzwords and super duper newest tech. They just ask how much, how exactly will it work, what are hidden cost, how will it be maintained and they will also look at your profile (clients, references, examples of finished projects etc).

lostcolony 6y ago

Failure is such a fun thing to think about, and it gets handwaved away so often. So many devs, architects, product owners, etc, just focus on happy path, and leave failure unspecced, unhandled, and just hope it never happens. And then boast about 99% uptime, but once you start questioning them you find out they get weekly pages they have to go investigate (and really the system is behaving weirdly a solid 10% of the time, but they don't know what to do about it and it eventually resolves itself, and they don't count "pageable weirdness" in their failure metric). It's actually one of the things I love about Erlang, and how it's changed my thinking. Think about failures. Or rather; don't. Assume they'll happen, in ways you can't plan for. Instead think about what acceptable degraded behavior looks like, how to best ensure it in the event of failure, and how to automatically recover.

deleted 6y ago
[deleted]
LargeWu 6y ago
As a developer who's trying to move on to either a management or product role, failure modes is one of the things I want to emphasize in that role. The sad fact is that so many product owners really don't understand how software works or gets built, and as such, they are unequipped to reason about such things.
jodrellblank 6y ago
On the subject of failures, you might like this blog post https://danluu.com/postmortem-lessons/ https://danluu.com/postmortem-lessons/ if you haven't seen it before.

adrianmonk 6y ago

See also Peter Deutsch's "Fallacies of Distributed Computing" list (https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...). There's some overlap, but also some new stuff. In particular, "pipes" isn't covered by the Fallacies list and is consistently a pain point and/or issue you always face in some way. Also "asynchronous" isn't covered by the Fallacies list.

wpietri 6y ago

Funnily, I thought the headline was talking about development process, as that also describes how a lot of places (mis-)handle the flow of what get worked on.

deleted 6y ago

[deleted]

evadne 6y ago

Do you have a moment to talk about our Lord and Saviour, Erlang/OTP?

shijie 6y ago
My thoughts exactly! As the Elixir code at my company continues to grow at a rapid rate, writing OTP services for everything has allowed me to never have to think about entire classes of bugs and edge cases simply by virtue of the patterns inherent in OTP and Elixir/Erlang. Right tool for the right job.

chubot 6y ago

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That is, most of a whole Linux distro. I have been loosely following some "git for binary data" projects for a number of years. I looked at IPFS like 5 years ago but it seems to have gone off the rails. The dat project seems to have morphed into something else? Are there any new storage projects in that vein? I think the OP is identifying a real problem -- distributed systems are unreliable, and you can get a lot done on a single machine. But we are missing some primitives that would enable that. Every application is littered with buggy and difficut network logic, rather than having a single tool like git (or rsync) which would handle the problem in a focused and fast way. It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue. Sort of related is a fast distro here I'm looking at: https://michael.stapelberg.ch/posts/2019-08-17-introducing-distri/ https://michael.stapelberg.ch/posts/2019-08-17-introducing-d...

trishume 6y ago
This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
- chubot 6y ago
  Yeah I agree, it's sort of an open problem, but I guess a bunch of arrows are pointing toward FUSE. Distri uses FUSE and it appears Microsoft's GVFS uses FUSE or whatever Windows technology is the equivalent. (My teammates developed Google's equivalent about 14 years ago, using FUSE, so it's something I've used / seen several times.) FUSE requires some kernel support (a module), which git of course doesn't require. That is a barrier, but perhaps not an insurmountable one. Basically I would like to offload all the "network" work to the OS, so applications are free of that logic.

perfunctory 6y ago

Just reading the title I assumed it was a post about business processes and communication between teams. Because this is how working for a big corp sometimes feel.

snazz 6y ago
It's a bit of Conway's law: the software structure mirrors the organization structure.

tlarkworthy 6y ago

I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It's how you outsource expertise. The vision of the service mesh is to make unreliability and security no longer the job if the application binary. Even without a service mesh, you can put a lot of common functionality into a reverse proxy. Personally I am loving OpenResty for the simplicity of writing adapters and oauth at the proxy layer with good performance.

trishume 6y ago
I think it should be possible to buy or use software from third parties. One thing I'm disappointed about and think we need better tools to avoid is the fact that third parties provide their tools as services rather than libraries. There's reasons they do that, deploying a library that all your customers can easily use is hard right now, but there's no reason it has to be that way. Some things can't easily be libraries like databases, but other things like some caches, miscellaneous operations like image resizing (depending on an architecture that can handle the load on those servers), and a bunch of other things could just be libraries.
- nemetroid 6y ago
  > Some things can't easily be libraries like databases The world's most widely deployed database engine (SQLite) is only available as a library.
  - tlarkworthy 6y ago
    The worlds most trafficked database is probably Google search* index and its not sqlite and it's certainly distributed. * Maybe Facebook, I dunno, the point stands.
    - 6y ago

gfxgirl 6y ago

doh! I thought this was going to be about remote work.

at_a_remove 6y ago

Yes, I have a little Python library for managing network shares in Windows. It has things like automatic retries that "back off" slowly, switching to cached IPs in case DNS is down, and checking to see if all of the drive-letters are full and either re-using a letter or creating a "letter-less" share. I had to develop it during a period of great instability within our network. It's ... large and over-engineered, but it just keeps on truckin'. On the other hand, it has been quite useful going forward, so that's a plus. I tend to program fairly defensively, in layers, right down to the much maligned Pokemon exception handling. The results don't have the, ah, velocity that is so often praised but they'll be there ticking along years later.

crazygringo 6y ago

Of course they do. But there's no alternative. No matter how fast or beefy your server is, these days if your product becomes a success, 99% of the time it will outgrow what's possible on a single server. (Not to mention needs for redundancy, geographic latency, etc.) And by the time you see the trend heading upwards so you can predict what day that will happen, you already won't have the time for the massive rewrite necessary. So yes, it's tons slower to write distributed servers/systems. But what other choice do you usually have? Though, as much as possible, you can try to avoid the microservices route, and integrate everything as much as possible into monolithic replicable "full-stack servers" that never talk to each other, but rather rely entirely on things like cloud storage and cloud database. Where you're paying your cloud provider $$$ to not fail, rather than handle it yourself. Sometimes this will work for you, sometimes it won't.

nixpulvis 6y ago
I've seen a handful of applications that attempt to "scale" by going down the micro-service route in a completely flawed way. Only to end up with a tangled mess that's impossible to reliably debug. All progress halts. There's nothing inherently wrong about your statement, just that it's still far to easy to write shitty distributed system, and so much easier to push that complexity off onto the OS or even network layer itself. Why should I the application developer care about the way my user's data enters my DB? This should be tightly abstracted away, and traced/logged accordingly. Leave it to me the systems developer to get the details right, and share the fruits of my labor with everyone. I can imagine no system more deserving of shared resources than network technology. Try and imagine a world without TCP/IP, do you not end up with something similar?
lowbloodsugar 6y ago
I think there are plenty of examples of teams that hit that cliff and had to rewrite. I am trying to remember specific articles. Counter-argument: I think stackoverflow hit a cliff, and remained monolithic, but that was several years ago [1] (sure they have a ton of servers, but they appear to be classic cache+web-server+database). There was another, maybe Instagram, that was monolithic and had to figure out how to shard their DB overnight. [1] https://nickcraver.com/blog/2016/03/29/stack-overflow-the-hardware-2016-edition/ https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha... Also, I don't think your post deserved a down-vote.

carapace 6y ago

> I hope this leads you to think about the ways that your work could be more productive if you had better tools to deal with distributed systems, and what those might be. We have tools. Promula/SPIN model checker is one just off the top of my head.

btreecat 6y ago

So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses. Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago. Then there are some folks/organizations building cathedrals, and the associated tooling (react, angular, maven, etc) and all the rest of us look up in awe and thing "well I guess if I want to be that good I need to use those tools on this dog house." But your dog house doesn't have the need to host parties, provide security, or even real weather protection other than a roof to keep the rain and sun out. Yet we all try to build our dog houses in ways that might be better if they are one day converted to a proper living quarters but likely will never have a need for running Water or windows.

colechristensen 6y ago
French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined with how the shell worked being driven by people who really really understood language and came up with one that made sense to them for interacting with computers... was, is still somewhat, wonderful. What's missing today is exactly what you mention. People designing tool sheds with materials often shoddily designed for cathedrals. Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. The favourite example of this is - perhaps apocryphal, but entirely believable - is replacing dozens of nodes using fancy big data tools with one node running sed/awk,etc. One thing is clear, nowhere I've been has had the tools readily available, the documentation clear and forthcoming, and the scale in the right range for projects. I found myself recently solving a problem with Hashicorp Vault using GCP to verify identity of machines wanting secrets. It was a stretch goal which had been on my plate for six months, every once in a while I would try to go back and figure out how to make it work, and months and months and months after trying, I put it together and it worked perfectly. The documentation to lead to this understanding had to be read out of order on several different pages with some lucky guesses to arrive at the solution, which in the end was just a few steps easily explained. Afterwards the documentation was fine and made perfect sense. Before I grokked the issue the documentation just seemed like a bunch on nonsense which led me to believe what I wanted to do wasn't possible in a constrained security environment. That is the kind of problem I solve all the time as someone with a decade of DevOps,SysAdmin,whatever experience behind me. Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating.

hn_throwaway_99 6y ago

Hah, I saw the title "Fragile narrow laggy asynchronous mismatched pipes kill productivity" and thought it was about the pitfalls of trying to coordinate remote teams across disparate time zones.

acjohnson55 6y ago
Yeah, my guess was it was a Slack takedown :)

ChrisMarshallNY 6y ago

This article has a point. But, as in all things software, "it depends." It depends on what the tools are, and what we are writing. In my own case, I have the luxury of writing fully native Swift code for Apple devices. I don't need to work with anything off the device, except for fairly direct interfaces, like USB, TCP/IP or Bluetooth. Usually. I have written "full stack" systems that included self-authored SDKs in Swift, and self-authored servers in PHP/JS. I avoid dependencies like the plague. Some of them are excellent, and well worth the effort, but I have encountered very few that really make my life as an Apple device programmer that much easier. The rare ones I do use (like SOAPEngine, or ffmpeg, for instance), are local to the development environment, and usually quite well-written and supported. If I were writing an app that reflected server-provided utility on a local device, then there's a really good chance that I'd use an SDK/dependency with network connectivity, like GraphQL, or MapBox. These are great services, but ones that I don't use (at the moment). I'm skeptical of a lot of "Big Social Media" SDKs. I believe that we just had an issue with the FB SDK. That said, if I were writing an app that leveraged FB services, I don't see how I could avoid their SDK. So I write fully native software with Swift, and avoid dependencies. That seems to make my life easier. But Xcode is still a really crashy toolbox.

nitwit005 6y ago

I've found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades or maintenance. Common example is data being pushed between systems with HTTP. Take the simplest case of propagating a boolean value. You toggle some setting in the UI, and it sends an update to another system with an HTTP request, retrying on a delay if it can't connect. This has two problems. The first is that if the user toggles a setting on and then off, you can have two sets of retries going, producing a random result when the far end can be connected to again. The second, is that the machine doing the retries might get rebooted, and people often fail to persist the fact that a change needs to be pushed to the other system. I've seen this issue between two processes on the same machine, so technically you don't even need a network.

jxcl 6y ago
If you're dealing with multiple processes persisting data to different data stores, you're dealing with a distributed database problem, which has many more failure modes than the ones you've described here. If there's anything I've learned across my career, it's to avoid distributed databases unless absolutely necessary, and if it is necessary, then spend a bunch of time trying to make sure you got it right. And then even after that you probably got it wrong.

robbrown451 6y ago

The title reminded me of the turboencabulator. https://www.thechiefstoryteller.com/2014/07/16/turbo-encabulator-best-worst-jargon/ https://www.thechiefstoryteller.com/2014/07/16/turbo-encabul...

jancsika 6y ago

> Untrusted: If you don’t want everything to be taken down by one malfunction you need to defend against invalid inputs and being overwhelmed. Sometimes you also need to defend against actual attackers. Sorry, but unless your centralized alternative is only used internally by troglodytes you have to at least defend against invalid inputs.

Andaith 6y ago

Just some fun with the English language: > probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed. If you don't ever plan on distributing your software you can save a _lot_ of time :)

emmelaich 6y ago

It's perfectly applicable to people too; one stickler for the rules or slow working in a critical role, say security officer, or change board chair can kill productivity.

ahh 6y ago

I feel so attacked right now.

lowbloodsugar 6y ago

Really like the curiosity and thought behind this article. Couple of thoughts: >probably upwards of 80% of my time is spent on things I wouldn’t need to do if it weren’t distributed. Sure. Do it on one giant machine. Then you'll be spending 80% of your time doing things you wouldn't need to do if it weren't monolithic. At the end of the day, if your customer is on the other end of the internet, then all of those complaints apply. If you solve that by running an app on their device, then oh boy are you going to have fun testing. I prefer scaling out. The stackoverflow peeps prefer scaling up. There are some great write-ups about how they scale. I found this [1] one after some quick googling, but I am certain there are more. So it's really about choose your poison. >I think people should be more willing to try and write performance-sensitive code as a (potentially multi-threaded) process on one machine in a fast language if it’ll fit rather than try and distribute a slower implementation over multiple machines. Sure. I once replaced a system that ran on 10 32-core machines with one that ran on one with four cores on one machine and did the work in the same time. Another time I had 96 cores, more threads, and I replaced it with one that had three threads and was faster. But both of those solutions were evolutionary dead-ends. The tasks were very specific, and not subject to change. The first one was a single C file. The latter was actually java, but with hand-rolled hash tables and optimistic locks. The first one I doubt I could follow it now. My point is, you can have understandable systems that good people (as opposed to geniuses) can work on, evolve and adapt, and that have well understood failure modes and scaling cliffs. Or you can have bonkers code that everyone is afraid to touch, and which fails in production when it hit a cliff you didn't know about and now your site is dead for eight days. If you can strike a good balance, then you'll probably have some combination of distributed, and brute force. [1] http://highscalability.com/stack-overflow-architecture http://highscalability.com/stack-overflow-architecture

naniwaduni 6y ago
> Sure. Do it on one giant machine. Then you'll be spending 80% of your time doing things you wouldn't need to do if it weren't monolithic. Deeply false equivalence.

_bxg1 6y ago

> I also think all these costs mean you should try really hard to avoid making your system distributed if you don’t have to. There's a point here about microservices.

lazyjones 6y ago

Networking and distributed software are well understood nowadays, the lower productivity of web companies vs. SpaceX etc. comes from all the unstable (both ever-changing and buggy) software they need to use. Most modern software is affected by this, but the web has it worse because of security issues and because the way browsers are evolving (on purpose, one has to add, because it's a cartel of large players on the web trying to stifle competition). SpaceX doesn't get some innovative new alloy they didn't order every couple of weeks and even the games industry has fewer obstructing external dependencies (hardware vendors and their drivers being one). At least that's my experience from about 20 years of web development (15 professionally).

steve76 6y ago

What's the opposite? "Sturdy broad quick harmonious complimentary pipes boost productivity" Details are inescapable, whether you are painting or composing or writing software. Ideas have holes in them and designs fall apart in your hands during implementation. The best approach I found is to just write down mistakes. I'm building software that reverses the freelance market. Instead of an endless list of buyers requests, there are dedicated stages buyers and programmers select, and everyone works together. Instead of tracking project goals, mistakes are track and there's specialized production stages to software design.

RandyRanderson 6y ago

TLDR he means microservices. We now have a generation that doesn't even recall a non-MS world. Tristan, the generation between us has created the IT world you describe. You'll probably spend the next 20 years of your career dealing with that mess. Sorry about that.