6 ms·
The Fallacy of Premature Optimization (2009)
- mpweiher 10y agoThe Knuth version goes on: "Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified." Above the famous quote: "The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by penny- wise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal; and I believe the same viewpoint should prevail in software engineering. Of course I wouldn't bother making such optimizations on a one-shot job, but when it's a question of preparing quality programs, I don't want to restrict myself to tools that deny me such efficiencies." All this from "Structured Programming with Goto Statements"[1], which is an advocacy piece for optimization. And as I've written before, we as an industry typically squander many orders of magnitude of performance. An iPhone has significantly more CPU horsepower than a Cray 1 supercomputer, yet we actually think it's OK that programs have problems when their data-sets increase to over a hundred small entries (notes/tasks/etc.). Anyway, I write a lot more about this in my upcoming book: "iOS and macOS Performance Tuning: Cocoa, Cocoa Touch, Objective-C, and Swift"[2] [1] http://sbel.wisc.edu/Courses/ME964/Literature/knuthProgramming1974.pdf http://sbel.wisc.edu/Courses/ME964/Literature/knuthProgrammi... [2] https://www.amazon.com/iOS-macOS-Performance-Tuning-Objective-C/dp/0321842847/ https://www.amazon.com/iOS-macOS-Performance-Tuning-Objectiv...
- mwcampbell 10y agoI'm surprised that anyone thinks it's OK when a program has trouble with a data set with over 100 small entries. Can you point to an example of that?
- jeyoor 10y agoIn some Sharepoint 2010 deployments I worked on, it was possible to create Sharepoint workflows that would would bog down, fail to process new entries, etc. once the list of items grew beyond 100-150 entries. Admittedly, this was probably related to misconfiguration and database issues (i.e. having zero oversight or administrative maintenance of the underlying MS SQL Server). That specific local minimum might not apply to the context of the article (optimization in code and systems design).
- quizotic 10y agoI've seen Hive take minutes (OMG!) to count a table with 5 rows ... but (other) people still think it's OK, because it scales well. It's latency sucks for small data sets, but it can handle very large data sets.
- Xorlev 10y agoIt's true, the startup costs of a MapReduce job are immense. I'm surprised by minutes, but I'm not sure this counts since there are and always will be different solutions and different tradeoffs for problems of different orders of magnitude. Any solution built for massive scale is often considers cumbersome for a small scale problem. For instance, I find test cases that spin up in-memory, in-process Spark extremely slow, but the spin up is quite fast overall in the context of a job that processes gigabytes of data per task.
- deleted 10y ago[deleted]
- chousuke 10y agoFrom what I've seen, the problem of inefficiency most often manifests when the entire system of software is taken as a whole. On a developer's machine with local SSD, a small database and a really small working set, even large inefficiencies can become invisible because the computer is just that fast. However, add in some network latency between the application and the database, some unpredictability about the timing of requests, and a much larger data set so that you won't hit data in memory 100% of the time, and suddenly code that was working fine grinds to a halt as it crosses a threshold where it can no longer process requests faster than they come in. Unfortunately, with modern libraries helping "abstract away" underlying architecture like databases and the network, code like this is very easy to write without even realising that there might be a problem until it hits you in production.
- SomeCallMeTim 10y ago>Unfortunately, with modern libraries helping "abstract away" underlying architecture like databases and the network, code like this is very easy to write without even realising that there might be a problem until it hits you in production. I think that modern library abstractions are a tip of the iceberg. There is entire copy-and-paste class of developer who doesn't understand what's going on under the hood with most anything they're doing. We're operating on a stack of a abstractions a dozen layers deep, and people rarely take the time to look more than one layer down. These folks are competent developers who can get things done. Even big things. But they are either incapable of envisioning how the entire system will interact, or they lack the appropriate functional details of how the abstractions work "behind the curtain" to be able to form an accurate mental model. As a result, they just wire things up without understanding how everything will interact. The Mythical Man Month talked about the "surgeon" role on a team. The programmer that's now known as the "10x" developer. That's what's needed for any project that will eventually need to scale, or you're almost certainly going to have major inefficiencies that make everyone's lives difficult for years. I'm actually considering marketing my consulting services as being that person for startups -- the architect you really need to design a solid foundation, so that you don't accidentally create a house of cards that will fall over the first time you hit the front page of HN. Not to say that anyone can foresee all interactions in a complex system. You still have to profile and test carefully. But I do consulting and have seen systems that had, to me, obvious bottlenecks that were poorly handled. One recommendation I made brought hosting costs for a small company down from $10k/month (and spiraling up out of control) to less than $500/month. And their product wasn't even that popular; it was not nearly at a Reddit, HN, or Quora level of usage. They peaked at about 2000 requests per second. I was handling more than that on a $10/month VM for my own backend (different load, and different requirements, but theirs could have been handled by a pair of app servers and a paid of database servers, two of each entirely for redundancy).
- doctorpangloss 10y agoAnd yet, the large organizations with R&D firms—making the most exciting greenfield software—almost exclusively focus their engineering efforts on squeezing the most software out of the lowest amount of hardware. They care deeply about every CPU cycle. You could argue that the fallacy of premature optimization has fallen many a line of business app. But if you're an engineer and you aspire for greatness, microoptimize. That's the rarer, harder-earned, harder-to-learn skill.
- dang 10y agoI always think of this comment which stuck in my mind years ago: "Consistently the hard technical problems are performance problems." https://news.ycombinator.com/item?id=1835859 https://news.ycombinator.com/item?id=1835859
- quizotic 10y agoAnd I've heard it said that lowering latency is harder than increasing throughput ... though whether it's more important probably depends on the context.
- SomeCallMeTim 10y agoI see this for * Machine learning * Computer graphics * Financial ...but not much else. The Facebook app is a nightmare of terrible design that has something like 100Mb of generated code to do something that should take a couple megabytes at most. And they are a large organization with R&D by any definition.
- doctorpangloss 10y agoTo your particular point, Facebook maintains multiple versions of its client, including a "basic mobile" version (https://mbasic.facebook.com https://mbasic.facebook.com) and a "touch" version (https://touch.facebook.com https://touch.facebook.com), among other idiosyncrasies between its various platforms. They put in a lot of engineering effort to make the best client for the appropriate device. With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs. But in terms of R&D, you could argue that Oculus is their major public R&D project. That project requires lots of performance optimization. Indeed, that is their greatest obstacle. Maybe you could call that computer graphics, but that's missing the point. Making the sensors and signal processors work on battery-powered hardware (the end goal) is really challenging. And then consider all the backend services that will serve these peripherals. This Premature Optimization Koan is really just a side effect of enumerative thinking. Like if you think about the problem hard enough, you'll be able to enumerate all of its aspects and select an appropriate solution. That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them. In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems. In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere. I'm sure at Facebook they share your concern. They probably work extremely hard to reduce the huge amount of code their iOS client is built from. I'm advocating that people train to improve performance, not to ignore the problem or dismiss it as "premature optimization," at least if they aspire to be engineers.
- pmoriarty 10y agoThere's an interesting talk by one of bzr's (formerly Bazaar's) lead developers on why bzr failed and lost to git.[1] In his view, much of the blame rested on the decision to defer optimization until the product was more mature. But by the time it was mature, key architectural decisions were made that were hard to change. Here's a salient quote: "Premature optimization" is a dangerous concept. It tends to imply you can defer all consideration of performance until you have the behaviour and UI you want. But, it turns out if the UI, core APIs or formats are hard or impossible to implement quickly, it is hard to unwind them. Now I'd have a more nuanced view of what "avoid premature optimization" means: something more like "avoid premature micro-optimization." The key question is: how expensive will it be to come back and change to a faster implementation later on, and how much does it save you to write a simple or naive one now. If there's one function or routine that is written in a straightforward way, going back to rewrite it should cost only whatever time it takes you to rewrite it, and then to test, debug and release that change. So it will often make sense to defer that work until you’re sure the requirements for the code really satisfy the user, your understanding of the problem is correct, there are no unknown bugs or edge cases, etc. However, there are some things that can make changing the code much harder and we ran into several of them... [1] - https://docs.google.com/presentation/d/1awg1CHM1w128iOBp_JOxE2DgHfywBeyjDe2bkx1vfVQ/edit https://docs.google.com/presentation/d/1awg1CHM1w128iOBp_JOx...
- brianwawok 10y agoExactly. Been on the losing side of this argument several times. Think a few if the products are unmaintainable piles now. Is mostly a lazy argument.
- mtkd 10y agoHaving seen teams try and fail to optimize late - I fully buy the concept it is never too early to optimize the architecture. If you get to the point you can't optimise because the fundamentals are too baked-in to realistically change - cut your losses that day and start pasting the usable parts in to a new project. It's important to differentiate between architectural optimisation (which should be optimised from the outset) and code optimisation. I do believe there is an argument for minimal code complexity (optimisation) in early phases - even if that leads to some latency - it will assist refactoring and onboarding new developers - later in the project team can revisit code that can benefit from being more complex.
- 4WIW 10y agoWhat can be worse than premature optimization? Premature generalization :-) What is so special about efficiency? Efficiency is just one of the concerns the software development orgs are facing, together with cost, utility, usability, deployability, maintainability and so on. The tradeoff between these (and the ability to choose this tradeoff correctly early in the cycle) often defines the success of a project or an organization. Focusing on one while disregarding others can be lethal. YMMV. Watch your controls.
- empath75 10y agoIt's interesting how much that reads like the exegesis of scripture, interpreting and reinterpreting quotes handed down by the prophets of some by gone age as if they're revealed wisdom. I don't think that it required all the verbiage about what Knuth said or what he really meant. Make the argument you want to make. Whether it's compatible with knuth or not is rather irrelevant to whether it's correct.
- Isamu 10y ago(I am in agreement with the article) Both Knuth and Hoare would never think that carefully choosing the optimal or near-optimal architecture or algorithm is premature. Unless the number of things you are processing is small and it doesn't matter what you do, you may as well do the easiest thing. Or, if the optimal thing is super hairy, are you sure it will be worth it? They are just arguing for a bit of common sense: identify the most important performance issues, and make sure you work on those. There are all kinds of optimizations that people bring up that are way down the list in terms of impact. 'Nuff said!
- esaym 10y agoWhether to optimize or not is still no excuse to write poor code or to put no thought in your code at all. I deal with both ends of this all the time :(
- munificent 10y agoThis is one of those classic statements that you only understand correctly after you have the experience it intends to impart. It teaches you nothing by itself. To really understand it, you need a clearer understanding of "premature" and "optimization". (Keeping track of the preceding text that clarifies that the quote is about small efficiencies doesn't hurt either.) If a performance choice affects the overall architecture, a public API, the user experience, or the success of your product in the market, optimizing now is not "premature". Now is exactly the time to get it right because if you don't, there might not be a later time to optimize. It is only premature if it taking the slow path doesn't matter much today and the technical debt won't accrue quickly such that it really matters in the future. If you can safely optimize it tomorrow, without it being significantly more costly to do so than it would be today, it's probably premature. Otherwise, it's probably not. Likewise, "optimization" doesn't just mean any code where you think about performance at all. You should always be aware of the performance of your code, just like you're aware of its maintainability, robustness, and test coverage. Performance is a fundamental aspect of the craft. Optimization is really about improving the performance of the code at the expense of other valuable things. Optimized code either takes significantly longer to develop (burning time you could have spent on other features or robustness), or is significantly harder to maintain (usually because the optimization relies on certain assumptions that you can no longer be flexible about) than non-optimized code. If the faster version doesn't take much longer to develop and doesn't lead to a more calcified codebase, then doing that isn't optimization. Not doing is just silly.
- erik14th 10y ago"build quickly to prove out a product feature, but always with a path to a more robust solution."[0] I like Discord's compromise. [0] https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7#.of26hgxom https://blog.discordapp.com/how-discord-stores-billions-of-m...
- combatentropy 10y agoIs that really why programs are slow: that programmers are misinterpreting a quote from Sir Tony Hoare? I thought it was because programmers tend to use fast computers, and they neglect to test on slow ones. Also they load their databases with test data that is much smaller than what the average user might have after using the program for a year or two.
- austincheney 10y agoEvidence of this: https://www.reddit.com/r/javascript/comments/5q9gxg/es6_feature_detection_shamelessly_copied_from/dcxh388/ https://www.reddit.com/r/javascript/comments/5q9gxg/es6_feat...
- xyzzy4 10y agoI always optimize my code for the minimal O(n) run time. From experience I've seen major benefits of doing this.
- andrepd 10y ago> Software engineers have been led to believe that their time is more valuable than CPU time; therefore, wasting CPU cycles in order to reduce development time is always a win. They've forgotten, however, that the application users' time is more valuable than their time. I've no words to express how much I agree with this. Developers who ship bloated software that chugs painfully even on high end computers, with the defense that "the developers time is more valuable than the CPU's", the developers who prefer to use brutally slower, but easier to use tools, they need to be reminded that this is their job. Your objective shouldn't be to make your own job easier. It should be to deliver the best product you can, and the best experience to the users. Not make them suffer long loading times, slow applications, unresponsiveness, and massive size bloat so you can write your shitty app in your breezy to use language. /rant
- quizotic 10y agoI've spent much of my career trying to optimize latency of long analytic-type queries; and I've been wrong on both sides of the issue. While it sounds stupidly tautological to say, I've come to think the important thing is being right in what I choose to optimize and when. With that bit of "wisdom" out of the way, there's a lot of ways the "premature optimization" phrase can be helpful: * I might think I know what's important/critical-path and be wrong (it's happened) * Users might be willing to live with slowness in some areas I didn't expect and demand performance in others * More performant solutions might come to light later * Particularly before product-market fit, features and their optimizations might be discarded * New development might change the critical path OTOH, I've found it's often beneficial to organize data and processing in a way that can take advantage of parallelism like multi-threading and vectorization. Often that's the kind of thing that's harder to do later.
- amai 10y agoAnd "Premature Flexibilization Is The Root of Whatever Evil Is Left": http://product.hubspot.com/blog/bid/7271/premature-flexibilization-is-the-root-of-whatever-evil-is-left http://product.hubspot.com/blog/bid/7271/premature-flexibili...
- amai 10y agoAnd "Premature Flexibilization Is The Root of Whatever Evil Is Left": http://product.hubspot.com/blog/bid/7271/premature-flexibilization-is-the-root-of-whatever-evil-is-left http://product.hubspot.com/blog/bid/7271/premature-flexibili...