4 ms·

> When Knuth wrote that quote, he meant something very specific with the word optimization: low level micro optimizations. Sure, but that isn't to say his advi

> When Knuth wrote that quote, he meant something very specific with the word optimization: low level micro optimizations. Sure, but that isn't to say his advice can't be applied in other contexts. Applying optimizations before knowing where a bottleneck is just a bad idea, regardless of the situation. To constrain the advice to the original context is sort of odd, but maybe there's more to it than that? I always thought the risk of premature optimization was similar to the lean principle of deferring commitment: you should to wait until the last possible moment to a decision that has high switching costs. In both cases you want to have as much information as possible to taking action. As an aside, this bugged me for a long time: how _much_ information do you need? Recently I co-presented with the CEO of a successful manufacturing company. At his firm they use the military's rule of thumb of taking action when you have 70% of the information. Deciding sooner than that and you risk making a bad decision; Waiting for more information after 70% and you likely are missing an opportunity.

hsn915 5y ago
The advice doesn't generalize to high level "optimizations". The problem with optimizations (low level) is they make the code more complicated and more difficult to understand. This is a problem. You better have really good payout. (It also takes a lot of time to apply micro optimizations to everything, so if you agonize about every CPU instruction you will never get anything done). But, general high level design decisions that don't produce slow code by default do not suffer from this. They don't make the code any complicated or hard to understand. Quite the opposite, they usually simplify things. Because most of the time, the way to get good performance is to do the minimal thing that needs to be done to perform the task at hand. The code you write will be very straight forward. There's no switching cost. On the other hand, choosing a slow language has a high switching cost. You can't just take a bad design and try to optimize it. It usually doesn't work. You have to rework the whole design. EDIT: I have one other important point to add: > Applying optimizations before knowing where a bottleneck is just a bad idea, regardless of the situation. Ignoring any and all performance concerns using this kind of reasoning will result in a codebase that is permeated with slowness. The program is slow but you can't identify any single bottleneck. You end up with a monster that simply can't be fixed. Worse, you may not even realize how slow it actually is. If you have profiled it and found no particular bottle neck, you may think this simply the limit of computer performance and the task at hand cannot be performed any faster than this.
- taeric 5y ago
  I love that you can replace "optimization" in your first paragraph with "abstraction" and have the same results. :D This actually leads me to see the concept of "micro abstractions", and I'm going to see if I see those in hard to deal with programs, now.
  - hsn915 5y ago
    I'm not sure what you mean, but with abstraction it's the opposite of what I said. Premature abstraction really is very very bad (I won't go so far as to say it's the root of all evil). The higher level the abstraction, the worse it is. Unless it's born from experience writing the same kind of program many times. In which case it's not "premature". "micro abstractions" are actually fine. You notice a recurring pattern in the code so you "abstract" it away into a function.
    - taeric 5y ago
      My guess is we are talking of different things when envisioning micro abstractions. I have in mind the projects that went from a handful of data structures and functions to dozens of each. Typically in the vein of abstracting away whatever immediate problem was faced in the code. Have a workflow? Make a workflow abstraction that can run any work. Have code that needs to retry? Make a retry facility that can retry for any reason. Than make a grammar to specify what backoff should be used. While doing those, realize you want a rules engine to determine what work and what backoff to use. Then, put in a sat solver. To determine if a plan can be done. ... None of these are really bad things, in and off themselves. Odds are high they would all be poorly done, or themselves a giant undertaking.
Jtsummers 5y ago
> the lean principle of deferring commitment: you should to wait until the last possible moment to a decision that has high switching costs. Last responsible moment, not last possible, moment. That's a big difference. You shouldn't start on the work at the last minute, that's too late. But you also don't (necessarily) need to start on something 2 years before it's due (unless it's big and/or there are anticipated problems that will need to be addressed that the time would permit; that's what makes it the last responsible moment).
- wiremine 5y ago
  Agreed, I should clarify that in my original comment.
xapata 5y ago
How do you know when you've had 7/10ths of "the information"? There's a great deal of scientific literature about this topic, BTW. Especially in machine learning.
dragontamer 5y ago
> Sure, but that isn't to say his advice can't be applied in other contexts. Applying optimizations before knowing where a bottleneck is just a bad idea, regardless of the situation. To constrain the advice to the original context is sort of odd, but maybe there's more to it than that? Donald Knuth writes entire volumes of books in assembly language, because the details matter. If you want to have that point, then you probably shouldn't be quoting Donald Knuth. Donald Knuth is the guy who points out that "for(int i=size-1; i>=0; i--)" is faster than "for(int i=0; i<size; i++)". When he says "don't sweat the small stuff", its because his books are filled with _incredibly_, detailed stuff. Far more detail than any other programming book / algorithms book. IIRC, the "premature optimization is the source of evil" statement was about how GOTO statements and how efficient they were. The __context__ is that GOTO is indeed a very fast construct (especially in the context of replacing recursion).
- Jtsummers 5y ago
  He made the comment a few times, apparently, in the context of GOTO it was this: > Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. From: "Structured Programming with Goto Statements", emphasis mine. Archive link: https://web.archive.org/web/20131023061601/http://cs.sjsu.edu/~mak/CS185C/KnuthStructuredProgrammingGoTo.pdf https://web.archive.org/web/20131023061601/http://cs.sjsu.ed... (I haven't read it in a while, and it's 41 pages, so I probably won't get to it today.)
- rvbissell 5y ago
  If one cared that much about speed, they'd use `--i`, not `i--`
  - dragontamer 5y ago
    Knuth wrote that discussion point in assembly language. I translated it to C so that people got the gist of it.

hsn915 5y ago

The advice doesn't generalize to high level "optimizations". The problem with optimizations (low level) is they make the code more complicated and more difficult to understand. This is a problem. You better have really good payout. (It also takes a lot of time to apply micro optimizations to everything, so if you agonize about every CPU instruction you will never get anything done). But, general high level design decisions that don't produce slow code by default do not suffer from this. They don't make the code any complicated or hard to understand. Quite the opposite, they usually simplify things. Because most of the time, the way to get good performance is to do the minimal thing that needs to be done to perform the task at hand. The code you write will be very straight forward. There's no switching cost. On the other hand, choosing a slow language has a high switching cost. You can't just take a bad design and try to optimize it. It usually doesn't work. You have to rework the whole design. EDIT: I have one other important point to add: > Applying optimizations before knowing where a bottleneck is just a bad idea, regardless of the situation. Ignoring any and all performance concerns using this kind of reasoning will result in a codebase that is permeated with slowness. The program is slow but you can't identify any single bottleneck. You end up with a monster that simply can't be fixed. Worse, you may not even realize how slow it actually is. If you have profiled it and found no particular bottle neck, you may think this simply the limit of computer performance and the task at hand cannot be performed any faster than this.

taeric 5y ago
I love that you can replace "optimization" in your first paragraph with "abstraction" and have the same results. :D This actually leads me to see the concept of "micro abstractions", and I'm going to see if I see those in hard to deal with programs, now.
- hsn915 5y ago
  I'm not sure what you mean, but with abstraction it's the opposite of what I said. Premature abstraction really is very very bad (I won't go so far as to say it's the root of all evil). The higher level the abstraction, the worse it is. Unless it's born from experience writing the same kind of program many times. In which case it's not "premature". "micro abstractions" are actually fine. You notice a recurring pattern in the code so you "abstract" it away into a function.
  - taeric 5y ago
    My guess is we are talking of different things when envisioning micro abstractions. I have in mind the projects that went from a handful of data structures and functions to dozens of each. Typically in the vein of abstracting away whatever immediate problem was faced in the code. Have a workflow? Make a workflow abstraction that can run any work. Have code that needs to retry? Make a retry facility that can retry for any reason. Than make a grammar to specify what backoff should be used. While doing those, realize you want a rules engine to determine what work and what backoff to use. Then, put in a sat solver. To determine if a plan can be done. ... None of these are really bad things, in and off themselves. Odds are high they would all be poorly done, or themselves a giant undertaking.

Jtsummers 5y ago

> the lean principle of deferring commitment: you should to wait until the last possible moment to a decision that has high switching costs. Last responsible moment, not last possible, moment. That's a big difference. You shouldn't start on the work at the last minute, that's too late. But you also don't (necessarily) need to start on something 2 years before it's due (unless it's big and/or there are anticipated problems that will need to be addressed that the time would permit; that's what makes it the last responsible moment).

wiremine 5y ago
Agreed, I should clarify that in my original comment.

xapata 5y ago

How do you know when you've had 7/10ths of "the information"? There's a great deal of scientific literature about this topic, BTW. Especially in machine learning.

dragontamer 5y ago

> Sure, but that isn't to say his advice can't be applied in other contexts. Applying optimizations before knowing where a bottleneck is just a bad idea, regardless of the situation. To constrain the advice to the original context is sort of odd, but maybe there's more to it than that? Donald Knuth writes entire volumes of books in assembly language, because the details matter. If you want to have that point, then you probably shouldn't be quoting Donald Knuth. Donald Knuth is the guy who points out that "for(int i=size-1; i>=0; i--)" is faster than "for(int i=0; i<size; i++)". When he says "don't sweat the small stuff", its because his books are filled with _incredibly_, detailed stuff. Far more detail than any other programming book / algorithms book. IIRC, the "premature optimization is the source of evil" statement was about how GOTO statements and how efficient they were. The __context__ is that GOTO is indeed a very fast construct (especially in the context of replacing recursion).

Jtsummers 5y ago
He made the comment a few times, apparently, in the context of GOTO it was this: > Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. From: "Structured Programming with Goto Statements", emphasis mine. Archive link: https://web.archive.org/web/20131023061601/http://cs.sjsu.edu/~mak/CS185C/KnuthStructuredProgrammingGoTo.pdf https://web.archive.org/web/20131023061601/http://cs.sjsu.ed... (I haven't read it in a while, and it's 41 pages, so I probably won't get to it today.)
rvbissell 5y ago
If one cared that much about speed, they'd use `--i`, not `i--`
- dragontamer 5y ago
  Knuth wrote that discussion point in assembly language. I translated it to C so that people got the gist of it.