7 ms·
Software would be 50% better if every developer understood Tesler's Law: "Complexity can neither be created nor destroyed, only moved somewhere else." The dri
by paulddraper 4y ago
Software would be 50% better if every developer understood Tesler's Law:
"Complexity can neither be created nor destroyed, only moved somewhere else."
The drive to simplify is virtuous, but often people are blind to the complexity that it adds.
Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain?
The correct solution depends on the circumstances. There are excellent uses of microservices. There are excellent uses of monoliths. There are excellent uses of monorepos. There are excellent uses of ... (wait never mind monorepos are just better).
Understand what is ESSENTIAL complexity and what is INCIDENTAL complexity. Simplify your system to remove as much incidental complexity as possible, until you are left with the essential and be satisfied with that.
- deterministic 4y agoNot true. You can make any system arbitrarily complex. And 95% of software developers IMHO are hell-bent on proving that true every single day. Micro-services is a GREAT example of this.
- zambal 4y ago> You can make any system arbitrarily complex. But isn't that introducing incidental complexity? Not sure if you actually disagree.
- ed_elliott_asc 4y agoDoes Tesler’s law apply to lines of code or architecture? I have absolutely seen complex code that was created (often for perceived “best practices” like DRY) which could be removed by simplifying the code.
- sebhans 4y agoI think it applies to problems, not solutions. The complexity of a given problem cannot change. If you try to ignore part of the inherent complexity of a problem (also called essential complexity) in your solution, it does not disappear but someone else must solve it somewhere else, or the problem is not really solved. If you build a solution that is more complex than the problem itself (in other words, if you add incidental complexity), this does not increase the complexity of the problem either, only the complexity of the solution. I think a good solution to any problem needs to match it in complexity. I regularly use this comparison as a benchmark for solutions. Of course, you can also see it this way: The complexity you remove from the code by making it cleaner is added to your team communication because you now have to defend your decision. (Only half joking.)
- codery 4y agoLast paragraph about essential and incidental complexity rings my ear about Rich Hickey's "Simple Made Easy" talk. Talk: https://youtube.com/watch?v=SxdOUGdseq4 https://youtube.com/watch?v=SxdOUGdseq4
- vinay_ys 4y agoBoth essential complexity and accidental complexity can be created or destroyed based on how functional scope or technical scope is defined and understood. In a live and evolving system, a lot of complexity is also because of not having a vision for how the product domain or market or ecosystem might evolve and having a coherent shared understanding of it amongst product managers and platform builders. When product direction flip-flops or technical choices flip-flops too widely, it creates lots of complexity lag/debt which can be very murky to clearly identify and attack.
- kqr 4y agoDo you have a suggestion on how to avoid the flip-flopping problem? Or is there a way to turn with the wind without increasing complexity? Genuinely curious.
- vinay_ys 4y agoWell, flip-flop is usually due to heavy investment too early – on poorly formed thesis of a potential product-market fit or business model (ie., vision). Early days has to be about optimizing aggressively for speed of learning/validation/formation of thesis. There has to be an overarching thesis that is well-formed (formed based on experience/tests from adjacent/related markets, key assumptions validated at reduced scale, strong backing by investors/founders etc). During this vision formation/validation stage, keep things very lightweight and don't over-invest in prolonged tech platforms. For example, if your product/service has an on-the-field operational part to it, then run that part with manual operations with pen/paper – pen/paper is extremely flexible and universally usable and survives all kinds of surprises in the field – don't wait to build a software solution to test your thesis in the field. Manual ops can scale quite well especially during learning phase (scale is not your goal, learning/validating is). Choose the use-case for your experiments carefully – optimizing for fast learning and applicability of that learning for next adjacent use-case you intend to expand to. Once you get going, still build the tech systems without dogmas or baking strong opinions in too much. Keep your engineers generalists and first principles problem solvers and interested in solving both tech and functional-domain problems and encourage them to be humble and curious – because both the functional and tech world is constantly changing. Don't hire a huge product management org – every engineer/manager should be thinking about customer/product first. Over time, parts of your product are more mature and parts of your product are very nascent and still volatile. If your entire team is still thinking customer/product first and build tech to solve problems from first principles, then they should have found the right coupling/cohesion balance between different parts of the system to avoid shared fate, high blast radius or high contagion of that volatility/instability affecting more mature parts of the system.
- lelanthran 4y ago> Complexity can neither be created nor destroyed, only moved somewhere else. I have to say that this short quote is not the whole story; for example it's ridiculously common for artificial complexity to be introduced into a system, like using microservices on a system that gets 1k users a day. In which case, it is sometimes possible to remove complexity, because you are removing the artificial complexity that was added.
- papito 4y agoI think that quote makes sense if you assume that the complexity is required. Over-engineering is a whole different topic. The problem is that the shops that move from monoliths to distributed systems are under the impression that all of their problems are now magically over "because microservices".
- simplotek 4y ago> (...) like using microservices on a system that gets 1k users a day. This sort of specious reasoning just shows how pervasive is the fundamental misunderstanding of the whole point of microservices. Microservices solve organizational problems, and their advantages in scaling and reliability only show up as either nice-to-haves or as distant seconds. Microservices can an do make sense even if you have 10 users making a hand full of requests, if those services are owned and maintained by distinct groups.
- lelanthran 4y ago> Microservices can an do make sense even if you have 10 users making a hand full of requests, if those services are owned and maintained by distinct groups. Maybe, but after the next CEO comes in, those groups would be reorganised anyway :-/ Few companies maintain their org chart for a large length of time. My last place had the microservices maintained by distinct groups when I joined. When I left a third of the people were gone and half the groups were merged into the other half. This is not an uncommon thing. Going microservices because you don't want to step on other peoples toes is a good reason, but it's an even bet that the boundaries will shift anyway.
- mrg3_2013 4y ago>>Understand what is ESSENTIAL complexity and what is INCIDENTAL complexity Succintly put!
- tracker1 4y agoI'd add that abstracting complexity should be done where it makes the rest of the system easier to understand over the abstraction. Too much abstraction can make systems harder to understand and work with instead of easier.
- kqr 4y agoAssuming you are not perfect, you must have implemented too much abstracted systems at some point. What went through your head when you did that? What thoughts legitimized that extra layer of abstraction that turned out to be superfluous?
- tracker1 4y agoI think the best example I can think of is implementing the Microsoft Enterprise Library Data Access Application Blocks. EntLib came out of MS consulting as a means of normalizing larger scale application development with .Net. DAB in particular entailed creating abstractions and interfaces for data objects and data access components. In practice in smaller projects, this made things far more difficult, as in general the applications only targeted a single database and weren't really doing automated unit testing at the time. It was kind of horrible, as VS would constantly take you to the IFoo interface definition instead of the DbFoo implementation when you were trying to debug. Not fun at all. It was much earlier in my career. Probably the most complex system I designed was an inherited permission interface down to a component level through a byzantine mess of users/groups and ownership for a CRM tool in a specific industry. If I had to do it today, it would be very, very different and much simpler. Also much earlier in my career (about 20 years ago). These days I've really enjoyed using scripting languages, mostly JS/Node and keeping things simple (I don't care for inversify for example). In general the simpler thing is usually easier to maintain and keep up with over time. Yeah, frameworks and patterns can make boilerplate easier but it's often at the expense of making everything else much harder. Vs just starting off slightly harder from the start, but everything is simpler overall over time. Aside, been enjoying Rust as well.
- murdoze 4y agoMicroservices reveal communication. I believe we do not have the right tools yet.
- simplotek 4y ago> Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain? The main misconception about microservices is that people miss why they exist and what problems they solve. If you don't understand the problem, you don't understand the solution. The main problems microservices solve are not technical, but organizational and operational. Sure, there are plenty of advantages in scaling and reliability, but where microservices are worth their weight in gold is the way they impose hard boundaries on all responsibilities, from managerial down to operational, and force all dependencies to be loosely coupled and go through specific interfaces with concrete SLAs with clearly defined ownerships. Projects where a single group owns the whole thing will benefit from running everything in one single service. Once you feel the need to assign ownership of specific responsibilities or features or data to dedicated teams, you quickly are better off if you specify the interface, and each team owns everything begind each interface.
- kqr 4y ago> Once you feel the need to assign ownership of specific responsibilities or features or data to dedicated teams, you quickly are better off if you specify the interface, and each team owns everything begind each interface. If team A needs a new endpoint from team B, what would a typical dialogue look like under microservices and a modular monolith, respectively?
- simplotek 4y ago> If team A needs a new endpoint from team B, what would a typical dialogue look like under microservices and a modular monolith, respectively?
- simplotek 4y agoHow teams interact is a function of the team/org, not the software architecture. What microservices easily provide in this scenario that is far harder to pull off with a monolith is that with microservices the service owners are able to roll out a dedicated service as a deliverable from that dialogue. Whether the new microservice implements a new version of the API or handles a single endpoint, the service owners can deploy the new service as an extension to their service instead of a modification, and thus can do whatever they wish to do with it without risking any impact on their service's stability.
- dimtion 4y agoGiven your last sentence incidental complexity can be created and destroyed (and is more difficult to destroy than create). The quote would probably be more accurate as: > "ESSENTIAL Complexity can neither be created nor destroyed, only moved somewhere else."
- kqr 4y agoEssential complexity can also be created and destroyed, though sometimes it happens earlier in the design process. Picking the problem you choose to solve is how you control essential complexity.
- makapuf 4y agoEssential complexity is inherent to the problem you have. The solutiom is layered between product design, technical design and implementation. What is essential complexity for a layer can be accidental for the layer above.
- LudwigNagasena 4y agoThat just makes it a tautology. It basically says “essential complexity exists”.
- peteradio 4y agoMaybe but it still useful to know that essential complexity exists and to identify it in your project.
- paulddraper 4y agoIt's a frame of mind. Often a developer will see something big or complex and see it as a problem. But they should consider whether this matches the size/scope of the problem being solved
- simplotek 4y ago> But they should consider whether this matches the size/scope of the problem being solved In professional software development projects, specially legacy projects, often times the complexity is not justified by the problem. There is always technical debt piling up, and eventually it starts getting in the way.
- philipLutz 4y agothere is no silver bullet
- hikawaii 4y ago[dead]
- AtlasBarfed 4y agoAren't microservices an example of complexity creation over the fundamental base case?
- sublinear 4y ago> Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain? The net gain was composability of microservices, distribution of computing resources, and the ability to marshall off implementation details. Just because those requirements were routinely ignored in the era of monoliths doesn't mean the complexity wasn't essential or didn't exist.
- rwmj 4y agoWe've been here before with object oriented programming which was supposed to introduce a world of reusable, composable objects, even connected by CORBA if you wanted to run them remotely.
- papito 4y agoThis is a very nice academic theory, but in real life you get half-assed, half-implemented hairballs that suck out your will to live if you have to untangle that.
- kqr 4y agoWhat are the sequence of events that happen in real life that take us from nice theory to hairballs, that academics fail to foresee?
- papito 4y agoMost companies severely underestimate the complexities of a distributed system and the work that goes into a truly resilient, scalable setup of this kind. An infrusctructfre of this sort is meant to solve very hard problems, not to make regular problems much harder.
- tracker1 4y agoThere's also the distribution of work... if one set of teams manages the deployments or communications issues between the microservices, while the micro-service developers can concentrate on the domain it can be a better distribution of work. Where as if the same teams/devs are going to do both sides of this, it may make more sense for a more monolithic codebase/deployment.
- m463 4y agoreminds me of that law: "For something to get clean, something else must get dirty" and the corollary: "You can get something dirty without getting anything clean"
- jack_riminton 4y agoHow does that apply here? (I’m not being facetious)
- m463 4y agoI was sort of thinking along the lines of... You must remove complexity to make understandable code. You can remove complexity without making anything understandable.
- hibikir 4y agoThe talk of essential and incidental complexity is, in practice, far less useful than it seems. It's just very easy to just agree in trying to just get rid of incidental complexity, in the same way that it's easy for people to agree on getting rid of unnecessary, wasteful government programs, until the second you define which ones you mean. Every time I've had that argument at work, ultimately there's no agreement of what is essential. In one of the worst cases, the big fan of the concept was a CEO considering themselves very technical, who decided that everything they didn't understand was obviously valueless and incidental, while everything they personally cared about was really essential complexity, and therefore impossible to simplify... even though the subsystem in question never had any practical applications until the company failed. So ultimately, either the focus on avoiding incidental complexity is basically a platitude, or just a nice way to try to bully people into your favorite architecture. A loss either way.
- cjonas 4y agoThe strict definition would be essential complexity is only the complexity that would be required to implement in an "ideal system" (eg no latency, memory concerns, etc). By that definition you cannot just abandon all incidental complexity (as our systems are no where near ideal). Instead, this way of thinking is helpful for keeping the essential complexity implementation isolated and free from the incidental complexity.
- SanderNL 4y agoThe disagreement is on what (sub)systems are even necessary/important. Heck, we usually have trouble agreeing on what goals we are achieving. Once we settle on what truly matters, separating the essential is not that hard.
- byyyy 4y ago>Complexity can neither be created nor destroyed, only moved somewhere else. Not true. I blow.up a car. Complexity is destroyed. I rebuild the car from blown up parts. Complexity is created. There's no underlying generic philosophical point about microservices and monoliths. What we can say is that microservices are not necessarily less complex than monoliths, but this relationship has no bearing on the nature of complexity itself.
- justinclift 4y ago> I blow.up a car. Complexity is destroyed. Wouldn't the blown-up car be more complex than the non-blown-up car?
- byyyy 4y agoIn that case complexity is created thereby proving my point anyway. Typically I define complexity as a low probability macrostate meaning low entropy. So debris generated from the explosion has a high probability of randomly occuring, while a new car has a very low probability of randomly occuring. Following this definition you arrive at a formal definition of complexity that is intuitive. Imagine a tornado that randomly reconfigures everything in it's wake. The more complex something is the less likely the tornado is going to reconfigure everything into that thing. So it is very likely for a tornado to reconfigure things into debris and extremely extremely unlikely for the tornado to reconfigure everything into a brand new car. A tornado reconfiguring atoms into a car seems to be impossible but it is not, it is simply a low probability nearly impossible configuration. Therefore the car is more complex then exploded debris. Think about this definition because it also aligns intuitively with the effort required and technical complexity of any object. The more effort and technical complexity it has the lower chance it has of a tornado randomly reconfiguring atoms to form that object. Thus that object is more "complex". Whatever your definition is, the quote saying complexity cannot be created or destroyed is fundamentally from any perspective usually not true. If you want to define it as simply microservices or monoliths it still doesn't make sense. Whose to say that when converting a monolith to microservices that complexity remains conserved? Likely the complexity rises or lowers a bit or a lot. Complexity doesn't remain the same even if you use informal and inexact fuzzy definitions of complexity.