3 ms·

This article seems goofy and weird. He spends a LOT of time randomly talking, in order to justify not using a profiler, when profiling is such a simple and easy

by jblow 12y ago

This article seems goofy and weird. He spends a LOT of time randomly talking, in order to justify not using a profiler, when profiling is such a simple and easy thing. I know many high-performance programmers and all of them profile because profiling is how you test your mental model against reality. Yes, as the author says, having a mental model of machine performance is important. But you need to test that against reality or you are guaranteed to be surprised in a big way, eventually. Example: How does he even know that his div optimization matters? If he is even reading through one pointer in that time, he is probably taking a cache miss on that read, the latency of which is going to completely hide an integer divide. The author seems generally to not understand this, since he spends most of his time talking about instruction counts. Performance on modern processors is mostly determined by memory patterns, and you can have all kinds of extra instructions in there and they mostly don't matter. Which this guy would know if he profiled his code.

chadaustin 12y ago
Hi Jon. I'm certainly familiar with caches and memory optimizations. I also know when I'm compute-bound (as in, the prefetcher is running ahead). Sorry if I wasn't clear - I love profilers! CodeAnalyst in particular is my go-to choice for "quick, I need a sample histogram across my functions". You're right that an example involving the memory subsystem would have been a good idea. My two points are: * It's possible to know something is on the latency critical path (e.g. div is ~20 cycle latency, but you run ~2 in parallel) without needing a profiler. Just look at the data flow through your algorithm. * When you begin an application, you should know your performance goals and approximately how you plan to hit them. If you end up building an application where you round-trip to the network six times to build your UI, you've just limited your best possible load time in Australia to over a second. That's all. :) p.s. I've never used that div optimization, though I think it's interesting.
- sqeaky 12y ago
  How do you know you are right if you didn't measure? I have done things "knowing" what the outcome would be only to be surprised, and I never would have known if I hadn't measured.
  - implicit 12y ago
    It's more about knowing your constraints ahead of time and working within them. If you're doing audio mixing, for instance, you probably have a thread that has to respond with samples within 1ms. Missing the window means catastrophic audio glitches. (it sounds terrible) It's a mistake to write this in Ruby and expect the profiler to tell you something you don't already know.
  - chadaustin 12y ago
    You have to measure at some point to build up your mental model. I run experiments all the time. In the specific example of buffer-builder, I have already built up a (reasonably accurate) mental model of modern CPUs, and I knew what I wanted the generated code to look like. Once I made the generated code look like I wanted, then I was not surprised to find that it outperformed existing libraries by 5x. :) I suspect the alternative approach, "profile the existing libraries and optimize hot spots" would have taken a lot more time.
    - sqeaky 12y ago
      With results that good it is surprising that you didn't document your benchmark procedures and include them in your article. I too feel comfortable working with modern CPUs but after performance sensitive projects I benchmark and/or profile to identify what I didn't know. How else can you learn (after listening to all the experts and reading all the documentation)? As for your feeling it would have taken longer with the "alternative approach", I must again ask for numbers. How do you know which approach would take longer without taking measurements on it? Is that with you taking that approach or an expert with that approach taking it. Are you an expert in that approach, yet humbly avoided stating so in the blog post? I don't really see them as alternatives. Using all the knowledge you have up front is simply a good design strategy, but once that knowledge is exhausted you can get more through testing empirically.
kllrnohj 12y ago
> in order to justify not using a profiler, when profiling is such a simple and easy thing. My day job is basically performance optimization. It is not simple nor is it easy to use a profiler. In fact, profilers often don't tell you anything useful at all. They'll tell you sort of where you're spending your time, but they don't help at all in telling you why you're spending your time there. Most of the major performance wins I find are from optimizing the architecture not from optimizing a hot loop. I can't even think of the last time I've seen any measureable difference from optimizing a hot loop. Heck, I can't even remember the last time I found a hot loop. Then again I work on things more like game engines and web browsers. Huge systems with tens of thousands of lines of code on the hot path. Profilers don't help all that much here, and I can optimize in advance in less time than it'll take to setup the profiler and "justify" the change.

by jblow 12y ago

chadaustin 12y ago

Hi Jon. I'm certainly familiar with caches and memory optimizations. I also know when I'm compute-bound (as in, the prefetcher is running ahead). Sorry if I wasn't clear - I love profilers! CodeAnalyst in particular is my go-to choice for "quick, I need a sample histogram across my functions". You're right that an example involving the memory subsystem would have been a good idea. My two points are: * It's possible to know something is on the latency critical path (e.g. div is ~20 cycle latency, but you run ~2 in parallel) without needing a profiler. Just look at the data flow through your algorithm. * When you begin an application, you should know your performance goals and approximately how you plan to hit them. If you end up building an application where you round-trip to the network six times to build your UI, you've just limited your best possible load time in Australia to over a second. That's all. :) p.s. I've never used that div optimization, though I think it's interesting.

sqeaky 12y ago
How do you know you are right if you didn't measure? I have done things "knowing" what the outcome would be only to be surprised, and I never would have known if I hadn't measured.
- implicit 12y ago
  It's more about knowing your constraints ahead of time and working within them. If you're doing audio mixing, for instance, you probably have a thread that has to respond with samples within 1ms. Missing the window means catastrophic audio glitches. (it sounds terrible) It's a mistake to write this in Ruby and expect the profiler to tell you something you don't already know.
- chadaustin 12y ago
  You have to measure at some point to build up your mental model. I run experiments all the time. In the specific example of buffer-builder, I have already built up a (reasonably accurate) mental model of modern CPUs, and I knew what I wanted the generated code to look like. Once I made the generated code look like I wanted, then I was not surprised to find that it outperformed existing libraries by 5x. :) I suspect the alternative approach, "profile the existing libraries and optimize hot spots" would have taken a lot more time.
  - sqeaky 12y ago
    With results that good it is surprising that you didn't document your benchmark procedures and include them in your article. I too feel comfortable working with modern CPUs but after performance sensitive projects I benchmark and/or profile to identify what I didn't know. How else can you learn (after listening to all the experts and reading all the documentation)? As for your feeling it would have taken longer with the "alternative approach", I must again ask for numbers. How do you know which approach would take longer without taking measurements on it? Is that with you taking that approach or an expert with that approach taking it. Are you an expert in that approach, yet humbly avoided stating so in the blog post? I don't really see them as alternatives. Using all the knowledge you have up front is simply a good design strategy, but once that knowledge is exhausted you can get more through testing empirically.

kllrnohj 12y ago

> in order to justify not using a profiler, when profiling is such a simple and easy thing. My day job is basically performance optimization. It is not simple nor is it easy to use a profiler. In fact, profilers often don't tell you anything useful at all. They'll tell you sort of where you're spending your time, but they don't help at all in telling you why you're spending your time there. Most of the major performance wins I find are from optimizing the architecture not from optimizing a hot loop. I can't even think of the last time I've seen any measureable difference from optimizing a hot loop. Heck, I can't even remember the last time I found a hot loop. Then again I work on things more like game engines and web browsers. Huge systems with tens of thousands of lines of code on the hot path. Profilers don't help all that much here, and I can optimize in advance in less time than it'll take to setup the profiler and "justify" the change.