12 ms·

My story on “worse is better” (2018)

195 points by rui314 4y ago

recursivedoubts 4y ago
The worse is better essay is a great read: https://www.dreamsongs.com/WorseIsBetter.html https://www.dreamsongs.com/WorseIsBetter.html Over time, I have come to believe that the problem is overly-aggressive abstraction. It is very tempting for most developers, especially good developers, to reach for abstraction as soon as things get complicated. And this can pay off very, very well in some cases. However, too much abstraction in a system leads to a very, well, abstract code base that becomes hard to get a handle on: there's no there there. You see this in the java world with AbstractFactoryBuilderLookupServiceBuilders and so forth, and with very elaborate type setups in functional programming languages. Concretizing the crucial bits of your system, even if that means a few large, complex and gronky methods or classes, ends up often making things more understandable and maintainable and, certainly, debuggable. John Ousterhout wrote a book that makes this point as well, advocating for "deep" rather than "shallow" classes and methods: https://www.goodreads.com/en/book/show/39996759-a-philosophy-of-software-design https://www.goodreads.com/en/book/show/39996759-a-philosophy...
- rob74 4y ago
  The kind of codebase you describe with "elaborate type setups" (multiple levels of inheritance, design patterns like Facade used for "decoupling" etc.) make it very hard to read the code and understand what it's doing (unless it's very well documented), and because of that also make it harder to extend without resorting to kludges. Or, if you want to extend it in a way the original designer didn't foresee, you now have to change 5 different classes to be able to access some private property in some class.
- funcDropShadow 4y ago
  > Over time, I have come to believe that the problem is overly-aggressive abstraction. Sometimes, yes, overly-aggressive abstractions are a problem. But the author describes policies of v1 and v2 of the linker. And I would say the most critical difference between them is, that the authors of v2 had a better understanding of the requirements that actually mattered. Therefore they were in a better situation to evaluate the architecture and they were able to make better trade-offs. Deciding to trust object files as inputs might raise a red flag for some people. In principle it could allow attacks/exploits of the linker. But in reality for most threat models this does not matter. Because the compiler generating object files has the same level of trust as the linker. Companies that want to provide an elf linker as a service product are out of scope. Deciding what is in scope and what is out of scope is probably one of the hardest decisions in product engineering. Because many of us software engineers lean towards perfectionism or, especially inexperienced developers are searching for the silver bullet, that set of rules that enables them to develop every product successfully. [edited typos]
- ncmncm 4y ago
  I suspect the "Worse is Better" essay has caused untold harm. It is written from the standpoint of a purist offended that the world doesn't appreciate purity, complaining that the things people end up using are built by pragmatic people. The lesson seems to be that doing things better is punished. But that is the wrong lesson. The correct lesson is that the real world is not obliged to conform to your personal model of "better". You have a personal obligation to continuously adapt your model to match the real world. This is the model of science, and is opposed to Platonism. After you have adjusted your model, it is certain to still not be right, and need further adjustment. Usually "the world" you are obliged to adjust to has its own problems. There are powerful forces making us favor accommodating a Microsoft execution environment, even though that execution environment has always been a cesspit. It represents its own poor abstraction. Posix file system semantics are another example. Von Neumann architecture and the C abstract machine are not the only, or best way to organize computational resources. It is important to recognize when somebody else's pragmatic failure threatens to taint your own models. Lisp, RG's hobbyhorse, didn't get sidelined because of Philistines. Lisp turned out not to be better, despite how strongly RG felt about it. Instead of figuring out what about Lisp was not right, he called things that were, along axes that matter, more right "worse", preserving his personal model and lessening readers' ability to reason about merit.
  - recursivedoubts 4y ago
    I don't agree at all. I think he was able to give a clear and accurate analysis of why lisp failed, despite liking the language so much. He also wrote critical responses to his own essay under a pseudonym, with a back and forth that is quite funny and demonstrates the ability to understand both sides of the argument. In "Worse is better" he explicitly mentions how that approach favors real-world application, because it is so simple it is fast and is "good enough" and then can be moved to 90% of the right thing. All of this is to say: I don't agree with you, but I also agree with you and I suspect he would as well, with qualifications. And he would probably also disagree with you.
    - fileeditview 4y ago
      Perfect user name! Also this was my comment of the day.. nay week at least!
- polotics 4y ago
  Also, I often see developers running to the abstractions they know (cough design patterns...) mostly because they're very afraid of acknowledging they haven't grasped the real complexity of a domain. By the time a pragmatic efficient design emerges, you're on the second or third rewrite...
- tynpeddler 4y ago
  One of the best programming aphorisms I've heard is "Everything in programming can be solved with another layer of abstraction except for the problem of to many layers of abstraction."
  - layer8 4y ago
    https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_softw...
- Bjartr 4y ago
  It can be worth understanding why the Java ecosystem evolved that way. It's not because simplicity wasn't valued, but rather it's not code simplicity that's valued. Rather, much of the Java ecosystem is designed to allow dozens of independent teams, totaling hundreds of developers, within a company (or across companies) to build their small piece of the application, with whatever build process is used by them. Then you put all the .jar files in one bundle, and the AbstractFactoryBuilderLookupServiceBuilders puts the pieces together at runtime. It's complex as hell code-wise, but it simplifies the amount of cross-team/cross-company alignment and synchronization that has to happen in order to cut a new version of the application containing hotfix 8495 for the part of the app maintained by Steve's team. There's actually a decent parallel between that and Microservices. Microservices make maintenance of the whole more complicated by introducing the network between pieces, but allow each piece more flexibility in how it's developed.
- robocat 4y ago
  From https://www.dreamsongs.com/WorseIsBetter.html https://www.dreamsongs.com/WorseIsBetter.html: The folks at Lucid were starting to get a little worried because I would bring them review drafts of papers arguing for worse is better, and later I would bring them rebuttals against myself. One fellow was seriously nervous that I might have a mental disease. after over a decade of thinking and speaking about it . . . [I wrote] "Back to the Future: Is Worse (Still) Better?" In this short paper, I came out against worse is better. But a month or so later, I wrote a second one, called "Back to the Future: Worse (Still) is Better!" which was in favor of it. This is the heart of engineering or politics: finding the optimal compromise.
- AtlasBarfed 4y ago
  Worse is better is simply a local maxima of features - schedule - cost tradeoffs, and there not being enough of a sustained investment to "hop out" of the local maxima. When those maxima are things like operating systems and programming language ecosystems, those are very very big hops to make.
hprotagonist 4y ago
There's some commonality here between this sort of "worse is better" and the observation that a meticulously neat and tidy, fastidiously clean {desk, notetaking system, editor configuration, ...} is a good indicator that its owner doesn't do anything worthwhile with it. The real world is messy. Things that come into contact with the real world will acquire a little bit of wear and mess. If it's all still brilliantly clean and tidy, you can't have done anything useful yet!
ncmncm 4y ago
Cf: "Architecture Astronaut". Abstraction always costs. This implies any abstraction you put in that doesn't deliver commensurate benefits makes your code fundamentally worse. If you have an abstraction with a name that seems to say it does X, but to be sure I have to trace through four other source files plus an unknown number of dead ends just to see if it really does exactly just X, and in the end it could have just been coded in place, it has already cost way more than any benefit it could yield. Abstraction is an engineering tool, not a moral imperative. You can always add abstraction later. So, "Worse is Better" is at best misleading. Better is better. But the measure of "better" you have been using is likely to be way off. People who think of themselves as smart tend to undervalue simplicity. It is a personal failing.
- KerrAvon 4y ago
  The flip side is that without abstraction your code becomes an unmaintainable, fragile disaster, leading to bad outcomes for the user. It’s not as simple in a complex system as you make it out to be.
  - ncmncm 4y ago
    Abstraction needs to be made to earn its keep. The massively abstracted linker in TFA was, exactly, an unmaintainable, fragile disaster. It was so bad they had to rewrite it from scratch. The new one certainly has abstractions of its own. Less harmful ones.
  - ratww 4y ago
    With way too much abstraction your code also "becomes an unmaintainable, fragile disaster, leading to bad outcomes for the user". Both extremes are terrible. And GP is not advocating using no abstractions.
- agentultra 4y ago
  It sounds like what you're talking about is indirection, not abstraction. If you have to verify that "it does X" by following source files and tracing execution then you're talking about indirection. An abstraction has mathematically sound laws that can be proven and introduce precise, new semantic layers to a program. By definition one doesn't have to think about the layers underneath the abstraction and can instead think entirely in the abstraction. The difficulty with abstractions is that few practising programmers know how to think even informally about abstractions. The mistake of substituting abstraction for indirection leads to the misguided notion that virtual methods and classes are "abstractions." However if you try to formalize these abstractions with relations and properties I think you will find most of these proofs difficult, if not downright impossible write. That's a good sign you don't have an abstraction.
chrchang523 4y ago
(2018)
- CalChris 4y ago
  The English translation of this essay was published in 2021.
0xdeadbeefbabe 4y ago
> It is OK to not aim to minimize the amount of code; reducing the complexity is much more important. Also, it's not OK to aim to increase the amount of code. //TODO include both statements in the creed.
CipherThrowaway 4y ago
IMO it's not that the simplest solutions are the best but that the "better" complex solutions are not actually available upfront. They can only be made with hard-won domain knowledge. The design policy for lld v1 per this article encoded many assumptions that turned out to be untrue in practice (like the importance of platform independence). If they had been true then the extra complexity might have been worth it. Over time the simpler lld v2 might accrue its own complexity that better reflects learned experience. Code is a tool for exploring and understanding problems as much as it is about solving them. Sophisticated solutions can't be designed before they are validated.
- duxup 4y ago
  My thing is I go simple. I'm good, maybe messy. Finally see enough / think I know a good abstraction and pull the trigger and later I find out ... oh man I was wrong. Knowing when you know is the hard part. I've got some abstractions out there I wrote early on when I didn't think I coded well and they have lived for years and saved tons of time. Others don't live long :(
  - lamontcg 4y ago
    https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction So many times I've taken code that was a mess because someone tried blindly DRY code for the sake of being DRY and rub abstractions on top of it and I've reverted the code back to being simple copypasta and its become so much more clearer and robust. Then you can look at the result and a simple abstraction may pop out which can reduce enough code duplication that the result is satisfactory (it may require a bit more boilerplate in the subclasses or whatever, but nothing likely to be brittle under future fixes).
    - bcrosby95 4y ago
      The nice thing about dealing with an overly-DRY codebase is you can unwind the abstractions. But an underly-DRY codebase... good luck. Copypasta tends to accumulate minor differences and figuring out if those minor differences matters can become very time consuming.
avgcorrection 4y ago
This ain’t “worse is better”. If the end-user has no worse of a user experience with the supposedly worse-is-better interface or program—most notably in this case if the linker never in practice gives a bad user experience on bad input (perhaps because it never happens)—then it’s not really “worse”. The central tenet of worse-is-better is to prioritize implementation simplicity over user experience. But if you simplify the implementation since some features are never used and not needed then you haven’t even had to make a choice on that spectrum—you have just cut out unneeded cruft. In fact the user experience has improved since it is faster...
- ncmncm 4y ago
  In fact the V1 implementation was not simpler. So, it was just worse on every axis except its adherence to what people apparently are taught in CS programs.
jt2190 4y ago
I’ve been thinking a lot about this kind of problem lately. My thinking was triggered by a description of real-world contracts by Lawrence Lessig, in a talk about crypto. [1] His point was that in most real world contracts there are many, many undefined contingencies for rare occurrences; the time and effort to nail-down what each party should do in those cases is just not worth it, and if any of them do actually occur everyone will go to court for adjudication. The software industry might be better off if we start making explicit that we have a real trade off of time and effort between designing a system that can handle all contingencies, and one that needs adjudication occasionally, i.e. throws an error or crashes, needs patching, etc. [2] Notice that Rui’s solution here was to basically allow lld to just crash under rare, extraordinary cases. This seems reasonable when written here, but in the real world suggesting that a system be allowed to crash, ever, often gets very hard pushback. [1] “Smart Contracts and DApps: Clarity versus Obscurity” https://youtu.be/JPkgJwJHYSc?t=3512 https://youtu.be/JPkgJwJHYSc?t=3512 [2] “Hard-assed bug fixin’” Joel on Software (2001)
- ncmncm 4y ago
  This is where the programming-language exception feature is valuable. You could crash, but you can also throw. Somebody somewhere sometime might be able to do a better job with the situation than you can right there and then. Checking status codes all up and down the call graph does not increase the likelihood that something could be done, but costs in the meantime. Hiding the checking behind abstractions does not reduce those costs.
  - jt2190 4y ago
    > You could crash, but you can also throw. Somebody somewhere sometime might be able to do a better job with the situation than you can right there and then. This is a very big assumption. Edit: The assumption is that this day can ever come. The author decided that in his system that day would likely never arrive.
    - ncmncm 4y ago
      There's no assumption, at all. Either it happens or it doesn't. If it doesn't, throwing an exception is a controlled sort of crash. There's no need for that person to decide, then and there, when they can just as well leave the choice to somebody else, at exactly zero cost.
scoutt 4y ago
I wouldn't called it "worse is better". Why not "less is better" or "simpler is better"? My take is that one should program in function of what the code should do, and not in function of what it's comfortable to me (as a developer). Yes, it's a great feeling when your code fits like a jigsaw puzzle, but also more complexity = more code being executed. Behind that RAII, behind that "operator=" and that "p = new Struct", etc. there might be extra complexity for the sake of developer's readability and comfortability. There is little or no added value for the end user or the purpose of the program itself. Also the code should be written "for the now", not for "that future feature it would be awesome to have someday like making it compatible with every other library X, etc.". At the end of the day, even without realizing it, your program is slow. I remember a developer where I work did a C# implementation of an AT command parser, in which every AT command was a separate DLL. It was very complex, and super slow. But the developer argued "if I need a new AT command, I'll just add a new DLL". It might have been better for him as a developer, but it was worse for the end user and the system in general. The code died the day that guy left the job.
- razorfen 4y ago
  > Also the code should be written "for the now", not for "that future feature it would be awesome to have someday like making it compatible with every other library X, etc.". The folksy software adage for this is YAGNI. (You Ain’t Gonna Need It)
- klik99 4y ago
  > I wouldn't called it "worse is better". Why not "less is better" or "simpler is better"? The original article called 'worse is better' was pretty successful and widely read, and part of it was the title grabbed people's attention. You could argue it's a rephrasing of "less is more" or "do one thing well" concepts for the click bait zeitgeist. That's not a bad thing - an old concept wrapped up in a new way of expressing can keep the idea alive - anyway it's funny and a little confusing, which might trigger someone to engage with it differently. There's many concepts that I've heard n times expressed in different ways only to have it click on the n + 1 way of phrasing it.
  - 4y ago
golergka 4y ago
> Here is the rule: if a user can trigger an error condition by using the linker in a wrong way, the linker should print out a proper error message. However, if the binary of an input file is corrupted, it is OK for the linker to simply crash. One of the most useful ideas in developing reasonably complex systems that I encountered is treating different types of errors differently. There's a place for both panics/exceptions on one hand result monads/error messages on another.
sam_lowry_ 4y ago
This echoes the "premature optimization is the root of all evil" saying.
draw_down 4y ago
phendrenad2 4y ago
Not a good example of "Worse is Better". This is "Worse is faster" which is much less interesting. The power of WIB is that is applies even if the programs are equally fast.
- renox 4y ago
  No, he also wrote that the V2 was easier to maintain..
  - phendrenad2 4y ago
    Still misses the point of the original worse is better paper.
CTmystery 4y ago
I have seen "worse is better" trotted out many many times, used mostly as an appeal to authority instead of a clear instantiation of the argument presented in the essay. I think this is because the argument in the original essay is not crisply presented (at least not to me). So authors take "worse" and "better" on some random dimension that's beneficial to them in the moment, and then appeal to the authority of this well respected essay to "prove" that their approach is better. This post is stating that they tried to generalize too much instead of building to narrow use cases first. There is no need to bring "worse is better" into it at all, IMO
- CRConrad 4y ago
  As I understand it, this all goes back to the original writings / theories / practices that "Worse is Better" was a reply to, which encouraged a "too theoretically pure" way of writing software: Everything abstracted away, generalised, (theoretically) infinitely flexible, etc -- at the cost of code complexity that in practice often turns out to be unnecessary for the vast majority of use cases. In that sense, every more or less radical simplification that sacrifices some of those fine (theoretical) advantages for simplicity can be said to be an example of "Worse is Better". Because what "Worse is Better" means is just that "theoretically 'worse', but much simpler code is actually better than theoretically 'better' but much more complex code". It's "worse" vs "better" on all those fine more or less theoretical dimensions against "worse" vs "better" on the single dimension of code complexity / simplicity. So I'd say this article, which was all about how a radical simplification of the code -- sacrificing the (unnecessary) generalization and flexibility they'd first tried to build into it -- turned out to bring a lot of other practical advantages, is a prime example of "Worse is Better". As I understand it.
nyanpasu64 4y ago
> Since the linker's input file is created not by humans but by the compiler, it is unlikely that the linker takes a corrupted file as an input. Therefore, the policy did not actually increase a crash rate. The code that trusts input was much simpler than the one that does not trust any byte of an input file. Interestingly I have encountered crashes in Ninja (not lld), caused by corrupted on-disk state I had to delete: https://github.com/ninja-build/ninja/issues/1978 https://github.com/ninja-build/ninja/issues/1978. I think I traced it down to a memory indexing or null pointer error, which would've been caught by asserts but they were disabled in release builds.
hardwaregeek 4y ago
What a lot of people seem to not understand is that mistakes or failings are sometimes on purpose. If software is slow, sometimes (not always!) it’s because the software that focused on performance didn’t get traction with users and failed. Too many programmers just see the immediate failings and not the larger, successfully avoided failings that the software prevented due to a tradeoff. Likewise many programmers don’t seem to get that the incentives of a programmer are quite different than the incentives of a manager. Programmers think “aw man my stupid manager is making me push out features instead of refactoring, if I were in charge I’d focus on code quality and performance”, not understanding that maybe, just maybe their manager might have different incentives and a different perspective. Worse is better is essentially a shorthand for understanding product management and scoping.
eikenberry 4y ago
> "It says that lazily-looking code that does not provide a consistent interface is sometimes actually better than neatly layered, consistent one." I find this an odd take on that paper. To me it has always been about how simplicity is more important than correctness. And while the authors take doesn't conflict with WiB, I do think they miss the point. > "Simplicity of implementation is very, very important, and you can sometimes sacrifice consistency and completeness for it." They almost get it in the conclusion. They are starting to see the important of simplicity, but still are fixed on correctness. Simplicity is not something you sacrifice correctness and completeness for "sometimes", it always wins if there is a contest. It is always more important (at least in line with WiB).
- CRConrad 4y ago
  > Simplicity is not something you sacrifice correctness and completeness for "sometimes", it always wins if there is a contest. Not quite: The absolutely simplest code is an empty code file. But that doesn't do anything when compiled, so I doubt it would become very popular; people want their software to do something. And if that "something" is equivalent to "rm -rf /", they'd prefer it to do something correct in stead. So code simplicity doesn't always win; having at least some darn somewhat-correct functionality easily beats it.
klysm 4y ago
> This may seem like an amateur-level programming mistake, but in reality, it's much easier to write straightforward code for each target than writing unified one that covers all the details and corner cases of all supported targets simultaneously. This is one of the reasons sum types are so critical in my opinion. They let you write code in that style, where OOP forces you to make everything look kinda the same.
kazinator 4y ago
> First of all, do many people really need a set of library functions and data structures that collectively work as a linker? Yes; people who unit test.
jstimpfle 4y ago
I've always considered the EINTR example in that article to be a bad one (although I think I agree with the article in general). Having blocking syscalls interrupted in case of an asynchonous signal is the right thing, because it allows the program to act on the signal. Think for example of a terminal user pressing Ctrl+C to interrupt blocking I/O to return to the shell prompt. The problem is that this does't go far enough - decades ago machines were running on a single CPU and OSes were focused on the scheduling of processes. Syscalls were all blocking, so for each individual process there could only ever be one syscall ("request") in flight at a time. Now, we're seeing a change (for example with io_uring) towards fully asynchronous I/O exposed to the userland, which allows submitting multiple requests to various I/O devices simultaneously, which has the potential to improve throughput a lot.
kazinator 4y ago
The "PC losering" anecdote in Gabriel's original essay is vert dated. In fact, neither design is the "better". In not-so-modern-anymore POSIX, you can choose whether a system call will be restarted after a signal is handled, or whether it will terminate with an error. Both requirements are needed. It is signals themselves that are "worse". But they let you have asynchronous behaviors without using threads. Sometimes you want a signal handler to just set some flag. This is because you have to be careful what you do in a signal handler, as well as how much you do. And then if you want the program to react to that flag, it behooves you to have it wake up from the interrupted system call and not go back to sleep for another 27 seconds until some network data arrives or whatever. In addition to sigaction, you can also abort a system call by jumping out of a signal handler; in POSIX you have sigsetjmp and siglongjmp which save and restore the signal mask. So that would be an alternative to setting a flag and checking. If you use siglongjmp, the signal itself can be set up in such a way that the system call is restarted. The signal handler can then choose to return (syscall is restarted) or bail via siglongjmp (syscall is abandoned). I wouldn't necessarily want to be forced to use siglongjmp as the only way to get around system calls being always restartable. Anyway, the Unix design showed to be capable of being "worse for now", and have space to work toward "better eventually". In the present story, the monolithic linker design isn't "worse". Let's just look at one aspect: crashing on corrupt inputs. Is that a bad requirement not to require robustness? No; the requirement is justifiable, because a linker isn't required to handle untrusted inputs. It's a tool-chain back-end. The only way it gets a bad input is if the middle parts of the toolchain violate its contract; the assembler puts out a bad object file and such. It can be a wasteful requirement to have careful contract checking between internal components. Gabriel naturally makes references to Lisp in the Rise of Worse is Better, claiming that Common Lisp is an example of better. But not everything is robust in Common Lisp. For instance the way type declarations work is "worse is better": you make promises to the compiler, and then if you violate them, you have undefined behavior. Modifying a literal object is undefined behavior in Common Lisp, pretty much exactly like in ISO C. The Loop macro's clause symbols being compared as strings is worse-is-better; the "correct requirement" would have been to use keywords, or else symbols in the CL package that have to be properly made visible to be used. I don't think that Gabriel had a well reasoned and organized point in the essay and himself admitted that it was probably flawed (and on top of that, misunderstood). The essays is about requirements; of course the assumption is that everyone is implementing the requirements right: "worse" doesn't refer to bugs (which would be a strawman interpretation) but to a sort of "taste" in the selection of requirements. Requirements have so many dimensions that it's very hard to know which directions in that space point toward "better". There are tradeoffs at every corner. Adopt this "better" requirement here, but then you have to concede toward "worse" there. If we look at one single requirement at a time, it's not difficult to acquire a sense of which direction is better or worse, but the combinations of thousands of requirements are daunting. If we look for what is the truth, the insight in Gabriel's essay it is that adherence to principled absolutes is often easily defeated by flexible reasoning that takes into account the context. 3.1415926 is undeniably a better approximation of pi than 3.14. But if you had to use pencil-and-paper calculations to estimate how many tiles you need for a circular room, it would be worse to be using 3.1415926. You would just do a lot of extra work, for no benefit; the estimate wouldn't be any better. Using the worse 3.14 is better than using 3.1415926; that may be the essence of "worse is better". On the other hand, if you have a calculator with a pi button, it would be worse to be punching in 3.14 than just using the button, and the fact that the button gives you pi to 17 digits is moot. A small bit of context like that can change the way in which the worse-is-better reasoning is applied.

195 points by rui314 4y ago

The worse is better essay is a great read: https://www.dreamsongs.com/WorseIsBetter.html https://www.dreamsongs.com/WorseIsBetter.html Over time, I have come to believe that the problem is overly-aggressive abstraction. It is very tempting for most developers, especially good developers, to reach for abstraction as soon as things get complicated. And this can pay off very, very well in some cases. However, too much abstraction in a system leads to a very, well, abstract code base that becomes hard to get a handle on: there's no there there. You see this in the java world with AbstractFactoryBuilderLookupServiceBuilders and so forth, and with very elaborate type setups in functional programming languages. Concretizing the crucial bits of your system, even if that means a few large, complex and gronky methods or classes, ends up often making things more understandable and maintainable and, certainly, debuggable. John Ousterhout wrote a book that makes this point as well, advocating for "deep" rather than "shallow" classes and methods: https://www.goodreads.com/en/book/show/39996759-a-philosophy-of-software-design https://www.goodreads.com/en/book/show/39996759-a-philosophy...

rob74 4y ago
The kind of codebase you describe with "elaborate type setups" (multiple levels of inheritance, design patterns like Facade used for "decoupling" etc.) make it very hard to read the code and understand what it's doing (unless it's very well documented), and because of that also make it harder to extend without resorting to kludges. Or, if you want to extend it in a way the original designer didn't foresee, you now have to change 5 different classes to be able to access some private property in some class.
funcDropShadow 4y ago
> Over time, I have come to believe that the problem is overly-aggressive abstraction. Sometimes, yes, overly-aggressive abstractions are a problem. But the author describes policies of v1 and v2 of the linker. And I would say the most critical difference between them is, that the authors of v2 had a better understanding of the requirements that actually mattered. Therefore they were in a better situation to evaluate the architecture and they were able to make better trade-offs. Deciding to trust object files as inputs might raise a red flag for some people. In principle it could allow attacks/exploits of the linker. But in reality for most threat models this does not matter. Because the compiler generating object files has the same level of trust as the linker. Companies that want to provide an elf linker as a service product are out of scope. Deciding what is in scope and what is out of scope is probably one of the hardest decisions in product engineering. Because many of us software engineers lean towards perfectionism or, especially inexperienced developers are searching for the silver bullet, that set of rules that enables them to develop every product successfully. [edited typos]
ncmncm 4y ago
I suspect the "Worse is Better" essay has caused untold harm. It is written from the standpoint of a purist offended that the world doesn't appreciate purity, complaining that the things people end up using are built by pragmatic people. The lesson seems to be that doing things better is punished. But that is the wrong lesson. The correct lesson is that the real world is not obliged to conform to your personal model of "better". You have a personal obligation to continuously adapt your model to match the real world. This is the model of science, and is opposed to Platonism. After you have adjusted your model, it is certain to still not be right, and need further adjustment. Usually "the world" you are obliged to adjust to has its own problems. There are powerful forces making us favor accommodating a Microsoft execution environment, even though that execution environment has always been a cesspit. It represents its own poor abstraction. Posix file system semantics are another example. Von Neumann architecture and the C abstract machine are not the only, or best way to organize computational resources. It is important to recognize when somebody else's pragmatic failure threatens to taint your own models. Lisp, RG's hobbyhorse, didn't get sidelined because of Philistines. Lisp turned out not to be better, despite how strongly RG felt about it. Instead of figuring out what about Lisp was not right, he called things that were, along axes that matter, more right "worse", preserving his personal model and lessening readers' ability to reason about merit.
- recursivedoubts 4y ago
  I don't agree at all. I think he was able to give a clear and accurate analysis of why lisp failed, despite liking the language so much. He also wrote critical responses to his own essay under a pseudonym, with a back and forth that is quite funny and demonstrates the ability to understand both sides of the argument. In "Worse is better" he explicitly mentions how that approach favors real-world application, because it is so simple it is fast and is "good enough" and then can be moved to 90% of the right thing. All of this is to say: I don't agree with you, but I also agree with you and I suspect he would as well, with qualifications. And he would probably also disagree with you.
  - fileeditview 4y ago
    Perfect user name! Also this was my comment of the day.. nay week at least!
polotics 4y ago
Also, I often see developers running to the abstractions they know (cough design patterns...) mostly because they're very afraid of acknowledging they haven't grasped the real complexity of a domain. By the time a pragmatic efficient design emerges, you're on the second or third rewrite...
tynpeddler 4y ago
One of the best programming aphorisms I've heard is "Everything in programming can be solved with another layer of abstraction except for the problem of to many layers of abstraction."
- layer8 4y ago
  https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_softw...
Bjartr 4y ago
It can be worth understanding why the Java ecosystem evolved that way. It's not because simplicity wasn't valued, but rather it's not code simplicity that's valued. Rather, much of the Java ecosystem is designed to allow dozens of independent teams, totaling hundreds of developers, within a company (or across companies) to build their small piece of the application, with whatever build process is used by them. Then you put all the .jar files in one bundle, and the AbstractFactoryBuilderLookupServiceBuilders puts the pieces together at runtime. It's complex as hell code-wise, but it simplifies the amount of cross-team/cross-company alignment and synchronization that has to happen in order to cut a new version of the application containing hotfix 8495 for the part of the app maintained by Steve's team. There's actually a decent parallel between that and Microservices. Microservices make maintenance of the whole more complicated by introducing the network between pieces, but allow each piece more flexibility in how it's developed.
robocat 4y ago
From https://www.dreamsongs.com/WorseIsBetter.html https://www.dreamsongs.com/WorseIsBetter.html: The folks at Lucid were starting to get a little worried because I would bring them review drafts of papers arguing for worse is better, and later I would bring them rebuttals against myself. One fellow was seriously nervous that I might have a mental disease. after over a decade of thinking and speaking about it . . . [I wrote] "Back to the Future: Is Worse (Still) Better?" In this short paper, I came out against worse is better. But a month or so later, I wrote a second one, called "Back to the Future: Worse (Still) is Better!" which was in favor of it. This is the heart of engineering or politics: finding the optimal compromise.
AtlasBarfed 4y ago
Worse is better is simply a local maxima of features - schedule - cost tradeoffs, and there not being enough of a sustained investment to "hop out" of the local maxima. When those maxima are things like operating systems and programming language ecosystems, those are very very big hops to make.

hprotagonist 4y ago

There's some commonality here between this sort of "worse is better" and the observation that a meticulously neat and tidy, fastidiously clean {desk, notetaking system, editor configuration, ...} is a good indicator that its owner doesn't do anything worthwhile with it. The real world is messy. Things that come into contact with the real world will acquire a little bit of wear and mess. If it's all still brilliantly clean and tidy, you can't have done anything useful yet!

ncmncm 4y ago

Cf: "Architecture Astronaut". Abstraction always costs. This implies any abstraction you put in that doesn't deliver commensurate benefits makes your code fundamentally worse. If you have an abstraction with a name that seems to say it does X, but to be sure I have to trace through four other source files plus an unknown number of dead ends just to see if it really does exactly just X, and in the end it could have just been coded in place, it has already cost way more than any benefit it could yield. Abstraction is an engineering tool, not a moral imperative. You can always add abstraction later. So, "Worse is Better" is at best misleading. Better is better. But the measure of "better" you have been using is likely to be way off. People who think of themselves as smart tend to undervalue simplicity. It is a personal failing.

KerrAvon 4y ago
The flip side is that without abstraction your code becomes an unmaintainable, fragile disaster, leading to bad outcomes for the user. It’s not as simple in a complex system as you make it out to be.
- ncmncm 4y ago
  Abstraction needs to be made to earn its keep. The massively abstracted linker in TFA was, exactly, an unmaintainable, fragile disaster. It was so bad they had to rewrite it from scratch. The new one certainly has abstractions of its own. Less harmful ones.
- ratww 4y ago
  With way too much abstraction your code also "becomes an unmaintainable, fragile disaster, leading to bad outcomes for the user". Both extremes are terrible. And GP is not advocating using no abstractions.
agentultra 4y ago
It sounds like what you're talking about is indirection, not abstraction. If you have to verify that "it does X" by following source files and tracing execution then you're talking about indirection. An abstraction has mathematically sound laws that can be proven and introduce precise, new semantic layers to a program. By definition one doesn't have to think about the layers underneath the abstraction and can instead think entirely in the abstraction. The difficulty with abstractions is that few practising programmers know how to think even informally about abstractions. The mistake of substituting abstraction for indirection leads to the misguided notion that virtual methods and classes are "abstractions." However if you try to formalize these abstractions with relations and properties I think you will find most of these proofs difficult, if not downright impossible write. That's a good sign you don't have an abstraction.

chrchang523 4y ago

(2018)

CalChris 4y ago
The English translation of this essay was published in 2021.

0xdeadbeefbabe 4y ago

> It is OK to not aim to minimize the amount of code; reducing the complexity is much more important. Also, it's not OK to aim to increase the amount of code. //TODO include both statements in the creed.

CipherThrowaway 4y ago

IMO it's not that the simplest solutions are the best but that the "better" complex solutions are not actually available upfront. They can only be made with hard-won domain knowledge. The design policy for lld v1 per this article encoded many assumptions that turned out to be untrue in practice (like the importance of platform independence). If they had been true then the extra complexity might have been worth it. Over time the simpler lld v2 might accrue its own complexity that better reflects learned experience. Code is a tool for exploring and understanding problems as much as it is about solving them. Sophisticated solutions can't be designed before they are validated.

duxup 4y ago
My thing is I go simple. I'm good, maybe messy. Finally see enough / think I know a good abstraction and pull the trigger and later I find out ... oh man I was wrong. Knowing when you know is the hard part. I've got some abstractions out there I wrote early on when I didn't think I coded well and they have lived for years and saved tons of time. Others don't live long :(
- lamontcg 4y ago
  https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction So many times I've taken code that was a mess because someone tried blindly DRY code for the sake of being DRY and rub abstractions on top of it and I've reverted the code back to being simple copypasta and its become so much more clearer and robust. Then you can look at the result and a simple abstraction may pop out which can reduce enough code duplication that the result is satisfactory (it may require a bit more boilerplate in the subclasses or whatever, but nothing likely to be brittle under future fixes).
  - bcrosby95 4y ago
    The nice thing about dealing with an overly-DRY codebase is you can unwind the abstractions. But an underly-DRY codebase... good luck. Copypasta tends to accumulate minor differences and figuring out if those minor differences matters can become very time consuming.

avgcorrection 4y ago

This ain’t “worse is better”. If the end-user has no worse of a user experience with the supposedly worse-is-better interface or program—most notably in this case if the linker never in practice gives a bad user experience on bad input (perhaps because it never happens)—then it’s not really “worse”. The central tenet of worse-is-better is to prioritize implementation simplicity over user experience. But if you simplify the implementation since some features are never used and not needed then you haven’t even had to make a choice on that spectrum—you have just cut out unneeded cruft. In fact the user experience has improved since it is faster...

ncmncm 4y ago
In fact the V1 implementation was not simpler. So, it was just worse on every axis except its adherence to what people apparently are taught in CS programs.

jt2190 4y ago

I’ve been thinking a lot about this kind of problem lately. My thinking was triggered by a description of real-world contracts by Lawrence Lessig, in a talk about crypto. [1] His point was that in most real world contracts there are many, many undefined contingencies for rare occurrences; the time and effort to nail-down what each party should do in those cases is just not worth it, and if any of them do actually occur everyone will go to court for adjudication. The software industry might be better off if we start making explicit that we have a real trade off of time and effort between designing a system that can handle all contingencies, and one that needs adjudication occasionally, i.e. throws an error or crashes, needs patching, etc. [2] Notice that Rui’s solution here was to basically allow lld to just crash under rare, extraordinary cases. This seems reasonable when written here, but in the real world suggesting that a system be allowed to crash, ever, often gets very hard pushback. [1] “Smart Contracts and DApps: Clarity versus Obscurity” https://youtu.be/JPkgJwJHYSc?t=3512 https://youtu.be/JPkgJwJHYSc?t=3512 [2] “Hard-assed bug fixin’” Joel on Software (2001)

ncmncm 4y ago
This is where the programming-language exception feature is valuable. You could crash, but you can also throw. Somebody somewhere sometime might be able to do a better job with the situation than you can right there and then. Checking status codes all up and down the call graph does not increase the likelihood that something could be done, but costs in the meantime. Hiding the checking behind abstractions does not reduce those costs.
- jt2190 4y ago
  > You could crash, but you can also throw. Somebody somewhere sometime might be able to do a better job with the situation than you can right there and then. This is a very big assumption. Edit: The assumption is that this day can ever come. The author decided that in his system that day would likely never arrive.
  - ncmncm 4y ago
    There's no assumption, at all. Either it happens or it doesn't. If it doesn't, throwing an exception is a controlled sort of crash. There's no need for that person to decide, then and there, when they can just as well leave the choice to somebody else, at exactly zero cost.

scoutt 4y ago

I wouldn't called it "worse is better". Why not "less is better" or "simpler is better"? My take is that one should program in function of what the code should do, and not in function of what it's comfortable to me (as a developer). Yes, it's a great feeling when your code fits like a jigsaw puzzle, but also more complexity = more code being executed. Behind that RAII, behind that "operator=" and that "p = new Struct", etc. there might be extra complexity for the sake of developer's readability and comfortability. There is little or no added value for the end user or the purpose of the program itself. Also the code should be written "for the now", not for "that future feature it would be awesome to have someday like making it compatible with every other library X, etc.". At the end of the day, even without realizing it, your program is slow. I remember a developer where I work did a C# implementation of an AT command parser, in which every AT command was a separate DLL. It was very complex, and super slow. But the developer argued "if I need a new AT command, I'll just add a new DLL". It might have been better for him as a developer, but it was worse for the end user and the system in general. The code died the day that guy left the job.

razorfen 4y ago
> Also the code should be written "for the now", not for "that future feature it would be awesome to have someday like making it compatible with every other library X, etc.". The folksy software adage for this is YAGNI. (You Ain’t Gonna Need It)
klik99 4y ago
> I wouldn't called it "worse is better". Why not "less is better" or "simpler is better"? The original article called 'worse is better' was pretty successful and widely read, and part of it was the title grabbed people's attention. You could argue it's a rephrasing of "less is more" or "do one thing well" concepts for the click bait zeitgeist. That's not a bad thing - an old concept wrapped up in a new way of expressing can keep the idea alive - anyway it's funny and a little confusing, which might trigger someone to engage with it differently. There's many concepts that I've heard n times expressed in different ways only to have it click on the n + 1 way of phrasing it.
- 4y ago

golergka 4y ago

> Here is the rule: if a user can trigger an error condition by using the linker in a wrong way, the linker should print out a proper error message. However, if the binary of an input file is corrupted, it is OK for the linker to simply crash. One of the most useful ideas in developing reasonably complex systems that I encountered is treating different types of errors differently. There's a place for both panics/exceptions on one hand result monads/error messages on another.

sam_lowry_ 4y ago

This echoes the "premature optimization is the root of all evil" saying.

draw_down 4y ago

phendrenad2 4y ago

Not a good example of "Worse is Better". This is "Worse is faster" which is much less interesting. The power of WIB is that is applies even if the programs are equally fast.

renox 4y ago
No, he also wrote that the V2 was easier to maintain..
- phendrenad2 4y ago
  Still misses the point of the original worse is better paper.

CTmystery 4y ago

I have seen "worse is better" trotted out many many times, used mostly as an appeal to authority instead of a clear instantiation of the argument presented in the essay. I think this is because the argument in the original essay is not crisply presented (at least not to me). So authors take "worse" and "better" on some random dimension that's beneficial to them in the moment, and then appeal to the authority of this well respected essay to "prove" that their approach is better. This post is stating that they tried to generalize too much instead of building to narrow use cases first. There is no need to bring "worse is better" into it at all, IMO

CRConrad 4y ago
As I understand it, this all goes back to the original writings / theories / practices that "Worse is Better" was a reply to, which encouraged a "too theoretically pure" way of writing software: Everything abstracted away, generalised, (theoretically) infinitely flexible, etc -- at the cost of code complexity that in practice often turns out to be unnecessary for the vast majority of use cases. In that sense, every more or less radical simplification that sacrifices some of those fine (theoretical) advantages for simplicity can be said to be an example of "Worse is Better". Because what "Worse is Better" means is just that "theoretically 'worse', but much simpler code is actually better than theoretically 'better' but much more complex code". It's "worse" vs "better" on all those fine more or less theoretical dimensions against "worse" vs "better" on the single dimension of code complexity / simplicity. So I'd say this article, which was all about how a radical simplification of the code -- sacrificing the (unnecessary) generalization and flexibility they'd first tried to build into it -- turned out to bring a lot of other practical advantages, is a prime example of "Worse is Better". As I understand it.

nyanpasu64 4y ago

> Since the linker's input file is created not by humans but by the compiler, it is unlikely that the linker takes a corrupted file as an input. Therefore, the policy did not actually increase a crash rate. The code that trusts input was much simpler than the one that does not trust any byte of an input file. Interestingly I have encountered crashes in Ninja (not lld), caused by corrupted on-disk state I had to delete: https://github.com/ninja-build/ninja/issues/1978 https://github.com/ninja-build/ninja/issues/1978. I think I traced it down to a memory indexing or null pointer error, which would've been caught by asserts but they were disabled in release builds.

hardwaregeek 4y ago

What a lot of people seem to not understand is that mistakes or failings are sometimes on purpose. If software is slow, sometimes (not always!) it’s because the software that focused on performance didn’t get traction with users and failed. Too many programmers just see the immediate failings and not the larger, successfully avoided failings that the software prevented due to a tradeoff. Likewise many programmers don’t seem to get that the incentives of a programmer are quite different than the incentives of a manager. Programmers think “aw man my stupid manager is making me push out features instead of refactoring, if I were in charge I’d focus on code quality and performance”, not understanding that maybe, just maybe their manager might have different incentives and a different perspective. Worse is better is essentially a shorthand for understanding product management and scoping.

eikenberry 4y ago

> "It says that lazily-looking code that does not provide a consistent interface is sometimes actually better than neatly layered, consistent one." I find this an odd take on that paper. To me it has always been about how simplicity is more important than correctness. And while the authors take doesn't conflict with WiB, I do think they miss the point. > "Simplicity of implementation is very, very important, and you can sometimes sacrifice consistency and completeness for it." They almost get it in the conclusion. They are starting to see the important of simplicity, but still are fixed on correctness. Simplicity is not something you sacrifice correctness and completeness for "sometimes", it always wins if there is a contest. It is always more important (at least in line with WiB).

CRConrad 4y ago
> Simplicity is not something you sacrifice correctness and completeness for "sometimes", it always wins if there is a contest. Not quite: The absolutely simplest code is an empty code file. But that doesn't do anything when compiled, so I doubt it would become very popular; people want their software to do something. And if that "something" is equivalent to "rm -rf /", they'd prefer it to do something correct in stead. So code simplicity doesn't always win; having at least some darn somewhat-correct functionality easily beats it.

klysm 4y ago

> This may seem like an amateur-level programming mistake, but in reality, it's much easier to write straightforward code for each target than writing unified one that covers all the details and corner cases of all supported targets simultaneously. This is one of the reasons sum types are so critical in my opinion. They let you write code in that style, where OOP forces you to make everything look kinda the same.

kazinator 4y ago

> First of all, do many people really need a set of library functions and data structures that collectively work as a linker? Yes; people who unit test.

jstimpfle 4y ago

I've always considered the EINTR example in that article to be a bad one (although I think I agree with the article in general). Having blocking syscalls interrupted in case of an asynchonous signal is the right thing, because it allows the program to act on the signal. Think for example of a terminal user pressing Ctrl+C to interrupt blocking I/O to return to the shell prompt. The problem is that this does't go far enough - decades ago machines were running on a single CPU and OSes were focused on the scheduling of processes. Syscalls were all blocking, so for each individual process there could only ever be one syscall ("request") in flight at a time. Now, we're seeing a change (for example with io_uring) towards fully asynchronous I/O exposed to the userland, which allows submitting multiple requests to various I/O devices simultaneously, which has the potential to improve throughput a lot.

kazinator 4y ago

The "PC losering" anecdote in Gabriel's original essay is vert dated. In fact, neither design is the "better". In not-so-modern-anymore POSIX, you can choose whether a system call will be restarted after a signal is handled, or whether it will terminate with an error. Both requirements are needed. It is signals themselves that are "worse". But they let you have asynchronous behaviors without using threads. Sometimes you want a signal handler to just set some flag. This is because you have to be careful what you do in a signal handler, as well as how much you do. And then if you want the program to react to that flag, it behooves you to have it wake up from the interrupted system call and not go back to sleep for another 27 seconds until some network data arrives or whatever. In addition to sigaction, you can also abort a system call by jumping out of a signal handler; in POSIX you have sigsetjmp and siglongjmp which save and restore the signal mask. So that would be an alternative to setting a flag and checking. If you use siglongjmp, the signal itself can be set up in such a way that the system call is restarted. The signal handler can then choose to return (syscall is restarted) or bail via siglongjmp (syscall is abandoned). I wouldn't necessarily want to be forced to use siglongjmp as the only way to get around system calls being always restartable. Anyway, the Unix design showed to be capable of being "worse for now", and have space to work toward "better eventually". In the present story, the monolithic linker design isn't "worse". Let's just look at one aspect: crashing on corrupt inputs. Is that a bad requirement not to require robustness? No; the requirement is justifiable, because a linker isn't required to handle untrusted inputs. It's a tool-chain back-end. The only way it gets a bad input is if the middle parts of the toolchain violate its contract; the assembler puts out a bad object file and such. It can be a wasteful requirement to have careful contract checking between internal components. Gabriel naturally makes references to Lisp in the Rise of Worse is Better, claiming that Common Lisp is an example of better. But not everything is robust in Common Lisp. For instance the way type declarations work is "worse is better": you make promises to the compiler, and then if you violate them, you have undefined behavior. Modifying a literal object is undefined behavior in Common Lisp, pretty much exactly like in ISO C. The Loop macro's clause symbols being compared as strings is worse-is-better; the "correct requirement" would have been to use keywords, or else symbols in the CL package that have to be properly made visible to be used. I don't think that Gabriel had a well reasoned and organized point in the essay and himself admitted that it was probably flawed (and on top of that, misunderstood). The essays is about requirements; of course the assumption is that everyone is implementing the requirements right: "worse" doesn't refer to bugs (which would be a strawman interpretation) but to a sort of "taste" in the selection of requirements. Requirements have so many dimensions that it's very hard to know which directions in that space point toward "better". There are tradeoffs at every corner. Adopt this "better" requirement here, but then you have to concede toward "worse" there. If we look at one single requirement at a time, it's not difficult to acquire a sense of which direction is better or worse, but the combinations of thousands of requirements are daunting. If we look for what is the truth, the insight in Gabriel's essay it is that adherence to principled absolutes is often easily defeated by flexible reasoning that takes into account the context. 3.1415926 is undeniably a better approximation of pi than 3.14. But if you had to use pencil-and-paper calculations to estimate how many tiles you need for a circular room, it would be worse to be using 3.1415926. You would just do a lot of extra work, for no benefit; the estimate wouldn't be any better. Using the worse 3.14 is better than using 3.1415926; that may be the essence of "worse is better". On the other hand, if you have a calculator with a pi button, it would be worse to be punching in 3.14 than just using the button, and the fact that the button gives you pi to 17 digits is moot. A small bit of context like that can change the way in which the worse-is-better reasoning is applied.