9 ms·
E. W. Dijkstra Archive: “Why is software so expensive?” (2009)
- thewarrior 11y agoWhile Dijkstra sits in his ivory tower , patiently figuring out the most elegant algorithm , in the startup world as Zuck says is all about moving fast , breaking things in order to find out whether we even need the algorithm in the first place. When my boss comes to me with the next story , I can't tell him that , "Hey , I think I need two weeks for this as the system is becoming overly complex. I think I need some time to refactor the system into something simpler and more reliable .. " I've seen an example of this first hand. We had an image editor that allowed users to manipulate and rotate layers using like you have in Photoshop. Instead of using straightforward affine transforms , they came up with some weird , extremely complex implementation I'm yet to get my head around .It isn't even documented as to how it works. Finally when the time came to make it behave differently , I reimplemented only the new behaviour using affine transforms and simple trig and routed the flow through the old code or the new one using flags. Not the most elegant solution but that's how it is in real world projects. It's only the output that matters. The internal complexity is not something the customers gives a shit about. They want their feature and they want it now. And they're not even sure what they want. So the need for rapid iteration to discover what you need to build in the first place , and the quest for the most ideal and mathematically elegant solution will always be in conflict with each other.
- jacquesm 11y agoDijkstra did a lot more for moving the programming needle than Mark Zuckerberg. Sure, Mark made more money, but that's not the only yardstick by which we measure achievement (fortunately!). Without a bunch of people doing basic research none of us would have jobs.
- thewarrior 11y agoI have great appreciation for people like Dijkstra and Knuth and the work that they do. They write operating systems , compilers and databases. But coming down from the ivory tower of academia , do you really need to go to the trouble of writing your custom CMS in Haskell after proving it's correctness in Coq ? As a further example , the succesful company KissMetrics built the first version of it's product in a month and shipped using SQLite (!!) as a DB. In their own words , they optimized 100 % for shipping speed. Software doesn't exist in a vacuum , it plugs into a business. The amount of rigor and research that must go into a piece of software depends on the domain and business requirements. To insist that all software be written to such a high standard is unrealistic and will never happen. I'm yet another software engineer who (most of the time) writes CRUD apps as quickly as possible so my boss pays me on time. I have no pretensions of being a 10x or being the hero who drags the world out of the dark pits of poorly specified and mathematically inelegant software with his superior intellect.
- bwindels 11y agoShipping a product with sqlite doesn't contradict with having well structured code. Also, it's fine to write a prototype to get to market faster. But building further on top of that prototype is dangerous, because it will become harder and harder to make changes to the system and still have it behave correctly. And once the incorrectness of the system gets beyond of what the business will accept, it's very hard not to do a full rewrite, which is often very dangerous as well. Granted, this is still what often happens ...
- ryderm 11y agoPart of being a good software engineer is knowing when to do the "wrong" thing (implement something poorly to ship fast). But you must be aware of the trade offs. For early stage companies/products that are struggling for a market it often makes sense to hack things, but at a company that has a large and growing customer base, it rarely does.
- 0xdeadbeefbabe 11y agoWell sure, but you can find your beliefs in ivory towers too. Also, argument is central to academia or my impression of it anyway.
- vezzy-fnord 11y agoBut coming down from the ivory tower of academia , do you really need to go to the trouble of writing your custom CMS in Haskell after proving it's correctness in Coq ? Dijkstra was nothing like this. In fact, he was probably one of the more practical academics (if that even makes sense to some people). He advocated top-down, logic-based methods for proving correctness that were based on incrementally transforming a high-level program specification into a final product fit for implementation. He likened it to composing music. He also deeply studied paradigms that were firmly imperative in nature. Besides his work on semaphores for concurrency control, his deductive system of predicate transformer semantics and the Guarded Command Language built on top are some of the most comprehensive notations for imperative languages. He was also on the team that developed ALGOL 60, so it's quite likely you owe most languages you've used partially to him. That and your conception of an academic using the analogy of the "Haskell CMS verified by Coq" is largely a straw man, though I can understand why you used it. HN is pretty crazy about these things, so it's easy to get caught up in the thought bubble thinking it's the norm.
- LunaSea 11y agoI don't think that the user you're answering to is criticising research perse but rather the irony of an academic talking about real world development scenarios.
- eterm 11y agoConsider that this essay is 33 years old and remains still relevant today such that we can read it and appreciate the issues. I wonder if Facebook will remain relevant for that long?
- philh 11y agoGiven that facebook isn't an essay - it provides different value to an essay, under different circumstances, to different people - I'm not sure what you're going with that question.
- PaulHoule 11y agoPart of the domain of application programming is about "agility" because, from the viewpoint of business, if they want something changed you should be able to do it fast. My experience is that agility is best served by building on top of a well thought-out framework. For instance, shops that use Ruby-on-Rails have answers to most of the common problems that turn up. If you're dealing with pre-framework applications in Perl, PHP, ColdFusion, Java and who knows what, the simple task of adding a field to a form involves touching code and schemas that are all over the place.
- blahblah3 11y agoOf course your boss and customers only care about the output. But if you need to make sure your output is correct or want to make it easier to add new things, a good design is crucial
- chucksmash 11y ago> The internal complexity is not something the customer gives a shit about. They want their feature and they want it now. None of this is false. At some point though, the internal complexity may grow to the point such that you can no longer deliver the features they want right now. What then?
- thewarrior 11y agoThat point is dangerously close. I don't now what to tell them . Not everyone appreciates the benefits of not adding ridiculous features and non stop changes that overcomplicate the system while not adding much value. In fact many of the features aren't even used. Also when a project is outsourced , then the outsourcing firm is only happy that there are lots of changes as that means more billable hours. They're not going to push back. And neither can the developers working there. This is yet another danger of completely outsourcing your project without any technical oversight. Hopefully , I'll soon be working on something else . ;).
- Dewie3 11y ago> I don't now what to tell them . You have to manage expectations. Not just do whatever it takes to get things working, until they don't work/can't be changed anymore. It all looks the same to the people who haven't taken a look at the plumbing themselves.
- crdb 11y agoThen the good developers responsible for most of the codebase leave for somewhere else using their experience as CV points, and the firm rotates through large numbers of inexperienced or incompetent engineers (who can't get a better job) as the team inexorably grows to several times its original size, whilst the number of new features dwindles until everybody is doing maintenance on the spaghetti ball, and existing features start breaking slowly.
- drzaiusapelord 11y agoI see you've also used BackupExec.
- nmrm2 11y agoYour example is a tad ironic. Why wasn't the original system implemented in the straightforward way? Unless there's a good reason, your example is a pretty effective demonstration of what Dijkstra is warning about.
- thewarrior 11y agoYes it is. But on the other hand, the system has worked all these years , thousands of customers used it and got their jobs done. At that time they needed quick prototypes to settle on a good design.
- logfromblammo 11y agoWhen it comes to translations and rotations, most of the algorithms out there use 4x4 matrices, dual quaternions, axis-angle rotations, or Euler angles. There may be some emerging work using geometric algebras. So when you say "extremely complex" I think quaternions, because i^2 = j^2 = k^2 = ijk = -1 . I once had the opportunity to refactor a lot of disparate implementations of 3D manipulations into a single math library, and migrate all the various algorithms to quaternion-based calculations. I actually had time to examine and understand a lot of different ways of doing the same thing and validate that they actually were all doing the same thing. But then we all got canned when the contract was used to wipe someone's backside. I think the former customer tossed all that custom code in the trash and decided to do everything with MATLAB. I hope that's working out well for them. And by that, I mean I hope they are put into a persistent vegetative state during a prison riot.
- thewarrior 11y agoHa ! See. It happens everywhere.
- GFK_of_xmaspast 11y ago"And by that, I mean I hope they are put into a persistent vegetative state during a prison riot" That's a little more invested in your work than i think is healthy.
- elsjaako 11y agoHe addresses this point: But apparently, many managers create havoc by discouraging thinking and urging their subordinates to "produce" code. Later they complain that 80 percent of their labour force is tied up with "program maintenance", and blame software technology for that sorry state of affairs, instead of themselves. So much for the poor software manager. (All this is well-known, but occasionally needs to be said again.)
- steveklabnik 11y ago> When my boss comes to me with the next story , I can't tell him > that, "Hey , I think I need two weeks for this as the system is > becoming overly complex. I think I need some time to refactor > the system into something simpler and more reliable .. " Sounds like you should blame Zuck, not Dijkstra.
- georgerobinson 11y agoWhat happens when the cruft becomes so overbearing that you can no longer deliver features? Not only can you no longer deliver features, but you now have to communicate that the state of the system is unworkable and you need X weeks to fix it. Thats a much more uncomfortable position to be in than say "Hey, to enable us to keep developing this system we need to allocate some time to refactoring. If we don't, there is a danger that the system becomes too internally complex too manage and starts to damage the reliability and flexibility of our product."
- aidenn0 11y ago99% of software won't be around that long.
- the_af 11y agoSorry, but my experience shows the opposite. I've worked in multiple jobs where some piece of software was an incredibly complex ball of mud, where things were left without refactoring because no-one currently in the company understands how they work, and no-one can spare the time to do the refactor properly. Where changing a method or property unexpectedly breaks some other, apparently unrelated feature. Needless to say, this made adding new features a pretty risky affair. This happens all too frequently to believe in your 99% statistic. For example, it happens with some pieces of software at my current job.
- aidenn0 11y agoIt happens all to frequently because the 1% tends to be very long lived. If 99% of babies died in the first year of life, all of your friends would still be more than one year old.
- markbnj 11y agoIt's a good piece, and Dijkstra's views on this remain relevant, even if the whole thing is ultimately an exercise in trying to build an ornate analogy that will somehow survive translation to other realms. Anyone who has been in software long enough to have tried to explain it to someone who isn't in software has had to go through the same basic process. That in itself ought to be somewhat enlightening. In any event, the article is worth the read if only for this: >> How do we convince people that in programming simplicity and clarity —in short: what mathematicians call "elegance"— are not a dispensable luxury, but a crucial matter that decides between success and failure? Amen, brother. Great question.
- crdb 11y agoI disagree. There's a realm of fields to which software engineering applies. There's fields where correctness, elegance and formal methods definitely matter; those involve complex products like airplanes (RIP, A400 victims...) or nuclear plant control systems. Or academia, which cannot progress with imprecise thinking. You can't run Facebook's operations the way you'd run a bakery's website's. But then there are heaps more fields where correctness absolutely does not matter, where a shiny cover over an awful spaghetti ball is enough, where you can lose half the client's data and say "sorry, crash" and he'll just nod and shrug, and where this is a lot cheaper and faster than hiring a professional and doing things properly, where the determinant of success is not the engineer, but the management team, their connections, and the sheer amounts of money that investors are willing to pump into the company to win the market (money they will mostly give only to the first competent person that pitches them the idea, giving the pretty-"MVP"-in-a-week team the edge over the formals). It's a trade-off between time and cost to market, and product quality; it won't change until the second way beats the first on the first two dimensions as well as the last.
- romaniv 11y agoThe problem is that software usually moves from the first realm to the second realm over time.
- 11y ago
- ExpiredLink 11y agoThe software 'industry' moved in the opposite direction.
- emilburzo 11y ago>> One of the problems caused by the non-understanding software manager is that he thinks that his subordinates have to produce code: they have to solve problems, and in order to do so, they have to use code. To this very day we have organizations that measure "programmer productivity" by the "number of lines of code produced per month"; this number can, indeed, be counted, but they are booking it on the wrong side of the ledger, for we should talk about "the number of lines of code spent". Amazing how we still haven't figured this one out yet. >> I know of one —very successful— software firm in which it is a rule of the house that for a one-year project coding is not allowed to start before the ninth month! In this organization they know that the eventual code is no more than the deposit of your understanding. It's so hard for me to imagine that such a software firm could actually exist. Even though I've noticed the advantages of thinking first and coding later on a personal project done in a more relaxed period, I guess it's just too hard to justify your cost without having deliverables (code commits, in this case).
- HenryTheHorse 11y ago> for a one-year project coding is not allowed to start before the ninth month! And here we are, asking team leads why development cannot begin till week 2 of a 24-week project. It is a really strange and funny industry we are in.
- psykovsky 11y agoI suspect managers and programmers might be counting the weeks from different points in time...
- aidenn0 11y agoI think at the level of complexity of current day products, the "think for the first 75% of the time" is flat out impossible. Certainly some of the complexity is incidental, but much of it is necessary for the sort of product we have today.
- blakehaswell 11y ago
- noblethrasher 11y agoWhy is the title labeled "2009"? I presume it's because that's when the transcription was last revised, but the actual essay was published in 1982; anything written by Dijksra cannot be more recent than 2002.
- aidenn0 11y agoThis was written in the late '70s, not 2009: https://www.cs.utexas.edu/users/EWD/indexChron.html https://www.cs.utexas.edu/users/EWD/indexChron.html
- stephengoodwin 11y agoMinor correction: This was publish in 1982[1] and was transcribed from a PDF in 2009[2]. [1] https://www.cs.utexas.edu/users/EWD/ewd06xx/EWD648.PDF https://www.cs.utexas.edu/users/EWD/ewd06xx/EWD648.PDF [2] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/EWD648.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...