7 ms·

> I dislike the “intuition doesn’t work, profile your code” mantra because it seemingly says profiling is a viable replacement for theoretical calculations, whi

• klysm 1y ago
  I think what he’s saying here is you can’t skip the basic math step to arrive at good performance. Staring at profiling results will lead you to a local minima
  • jmull 1y ago
    Yeah, that's why I think it's nonsense. Measuring doesn't mean don't think. Measuring and thinking are two different things. You need to do them both to optimize effectively.
  • Joel_Mckay 1y ago
    Or optimizing a scheduling noop loop dozens of times given local encapsulation obscures global functional context. The fact remains most projects that do small trivial modular prototypes first will ultimately know which paths are viable before painting themselves into a corner algorithmically. Best of luck =3
  • deleted 1y ago
    [deleted]
  • cogman10 1y ago
    Profiling doesn't mean you don't do the math. Profiling is simply there to show you that "hey, this is where the problems actually are". You do profiling because it's WAY too easy to get obsessed about theoretical problems when a simple measurement will show you the actual problems. You do the math on the actual problem location, not a method with O(n!) which only gets called with n=3. You still have to look at the entire call stack when profiling (which means thinking about the overarching algorithm).
  • absolutelastone 1y ago
    I think the person you are responding to is criticizing the original "mantra" quote not the author's criticism of it(?)
  • 9rx 1y ago
    Well, he's saying that intuition does work... But does it really? If a problem area is so intuitively obvious, why would you introduce the problem in the first place? In reality, performance optimizations are usually needed where you least expect them. Which means that you can't get there intuitively. Hence, the suggestion of using profiling to help track down where the problem is instead.
    • mystified5016 1y ago
      Do you want to claim you've never written quick and ugly code to get something working to come back and fix it up later? Pretty much everyone I know will throw down an O(n^2) algorithm or whatever in their first pass and replace it with something more thought out once they have the time to think deeply about it. If you're fretting about optimization at every stage of development, you're really doing it wrong. This is precisely the type of early optimization that everyone and their dog will tell you is bad. You should focus first on making your program work and laying out the logic and data flows. Optimization does not matter until the program is almost finished.
      • tough 1y ago
        One could even say optimization does not matter until the program is already running.
      • deleted 1y ago
        [deleted]
      • deleted 1y ago
        [deleted]
      • 9rx 1y ago
        Purposefully introducing underperforming code as a tradeoff to meet some other goal (faster delivery, for example) is something else entirely. It doesn't require intuition or profiling – at most requiring only memory of the choice. A fun aside, but not related to what we're talking about.
      • patrick451 1y ago
  • deleted 1y ago
    [deleted]
  • bqmjjx0kac 1y ago
    FYI, the author is a woman.
• titzer 1y ago
  No amount of measuring and squeezing--not even years of it--is a subsitute for high-level thinking. And vice versa. Imagine: function F() { for (i = 0; i < 10; i++) { A(); B(); C(); } } If we profile this code, we might find out, e.g. B takes the majority of the time--let's say 90%. So you spend hours, days, weeks, making B 2X faster. Great. Now you removed 45% of execution time. But the loop in the outer function F is just a few instructions, it is not "hot"--it won't show up in profiles except for ones that capture stacks. If you're just stuck in the weeds optimizing hot functions that show up in profiles, it's possible to completely overlook F. That loop might be completely redundant, causing 10X the workload by repeatedly computing A, B, and C, which may don't need to be recomputed. There are bazillions of examples like this. Say you find out that a function is super, super hot. But it's just a simple function. There are calls to it all over the code. You can't make it any faster. Instead you need to figure out how to not call it at all, e.g. by caching or rethinking the whole algorithm. > How could measuring be a substitute for thinking/analyzing/predicting/forming a plan? This happens more than you think. Understanding how the system works in enough detail and also at a high level to formulate a plan is in short supply. Jumping in and hacking in things, like a cache or something, is surprisingly common.
  • dmurray 1y ago
    > it won't show up in profiles except for ones that capture stacks I don't think I've ever used a profiler that couldn't report you were in F() here. One that only captures your innermost functions really doesn't seem that useful, for exactly the reasons you give.
    • cogman10 1y ago
      The default usage of perf does this. There's also a few profilers I know of that will show the functions taking the most time. IMO, those are (generally) nowhere near as useful as a flame/icicle graph. Not saying they are never useful; Sometimes people do really dumb things in 1 function. However, the actual performance bottleneck often lives at least a few levels up the stack.
• kccqzy 1y ago
  I think theoretical calculations could mean a detailed calculation of the number of times a certain function or operation is called. We all know from tech interviews that there are big O time complexity, but this is usually very hand-wavy and not precise enough. You can usually come up with a more precise formula though it can get messy with recurrences if your algorithm involves recursion. You probably need a computer algebra system. Spending an afternoon doing these symbolic calculations might give you better intuition of why the profiling produced such a measurement.
• fabian2k 1y ago
  While profiling and measuring is very important if you want to optimize performance, there are a lot of things you can do without any profiling. In many situations the consequences of each approach are well known or easily reasoned about. Most of the time it's simply "do less work" = "more performance", or avoiding obvious and well-known patterns like N+1 database queries. But it's also very easily to mislead yourself that way, many "optimizations" might do much less than you think. So you should avoid implementing more complex or harder to understand code just because you think it is faster, but otherwise I'd certainly try to write faster code by default in areas I know well enough to judge that.
• jandrewrogers 1y ago
  I think many people have seen both sides of this in practice. I’ve seen engineers follow the profiler into a dead-end because they see nothing that stands out in the profiler or they don’t grok the essential nature of the code or system they are profiling. I’ve seen engineers with deep domain expertise consistently make accurate estimates of how a code changes will impact performance without ever using a profiler because their mental model of the code execution maps to reality with high fidelity. Profilers fill in gaps in our mental model of code execution but they are not a substitute for it. Computer behavior is largely knowable from first principles, albeit requiring a considerable degree of expertise and detail. For some people in some contexts, the profiler adds little information to what they already understand. I know a few people that do almost all of their optimization work using pencil and paper with great effectiveness and precision. Not something I would recommend for most software engineers but not unreasonable at the limit either.
  • infogulch 1y ago
    Optimize according to your mental model; profile to check that your mental model matches reality. If you optimize infrequently, or haven't profiled code like this recently, or haven't profiled this specific codebase recently, then your mental model is probably due a refresh.
  • kaptainscarlet 1y ago
    A lot of the optimisation I do largely consists of a series of highly educated guesses and the resulting fixes are right in 99% percent of the cases.
  • YZF 1y ago
    In a complex system it isn't always easy to reason about the behavior. Even if you fully understand the cpu, the compiler, the kernel, the disks etc. the way they interact in a real application isn't easy to hold in your head or to deal with using theoretical tools. I've spent a lot of time optimizing video codecs and while it is maybe possible to reason about these without a profiler it's much faster to profile. The profiler isn't going to tell you what you need to do though.
• hinkley 1y ago
  I think the confusion sneaks in because “measuring” is something you do several times and each iteration has different connotation. Once you “find” a candidate change you measure it to see if what you did made things worse and you put it back if it did, or maybe you try it in combination with other changes to see if its value is complementary. But people fuck up all the time reading the initial telemetry, which is often where I come in. I get tired of hearing people say, “we’ve done all we can, look how flat this chart is,” and hand someone my beer. You won’t find all of the good candidates in the percentage of run time list. That’s not all the data that’s there, and not every change that works needs to even be supported by the initial data. It only needs to be supported by the delta afterward.
• AtNightWeCode 1y ago
  Agree. A classic example is compilers that let you choose between optimizing for speed or binary size. But a smaller sized binary is sometimes faster.
  • tough 1y ago
    Why not both?
    • AtNightWeCode 1y ago
      Impossible. Speed option may do things like loop unrolling, function inlining and today even way more complicated things than that and therefor creates larger binaries.
    • touisteur 1y ago
      One way to improve performance is to unroll loops and inline code. Unfortunately this increases code size and puts pressure on the instruction cache, making a program sometimes slower. It's probably a lot harder to balance these out in the compiler than to just... sometimes try.
• tmoertel 1y ago
  > > I dislike the “intuition doesn’t work, profile your code” mantra because it seemingly says profiling is a viable replacement for theoretical calculations, which it isn’t. > This seems like a nonsensical statement to me. How could measuring be a substitute for thinking/analyzing/predicting/forming a plan? I believe the mantra “intuition doesn’t work, profile your code” is understood to mean “don't rely on your intuition alone; profile your work.” It's a warning that when it comes to performance optimization, intuition is often unreliable, so you need to supplement your mental model with actual data if you want to avoid big mistakes. I don't know why the author of the blog post is interpreting the mantra literally, as if it claims intuition is obsoleted by profiling.
• purplesyringa 1y ago
  You're right, I could've phrased that better. Profiling to find suboptimal code is perfectly fine. Then you need to figure out how to fix it. Many people don't understand how performance optimization works, so they blindly add caching, improve constant time by invoking more low-level methods, etc. This obviously doesn't work, yet intuitively (to those people, anyway) it should produce good results. That's why the mantra exists: don't trust your intuition, don't believe it when it says these changes improve performance, instead measure that performance and only apply changes that work. This is also perfectly fine, but this is a double-edged sword, and I've seen people go too far in this direction. For example, they refuse to do any changes that don't immediately improve performance according to the profiler. If they modify one computation and performance decreases, they abandon this path altogether. They treat optimization as a game with a dense fog of war, and they refuse to apply deductive reasoning and, of course, intuition to apply changes that, according to the profiler at least, are not immediately rewarding.
  • Capricorn2481 1y ago
    Sounds like a tricky balancing act. There are things that are extremely difficult to "game out." CPUs are very complicated. There are optimizations that seem like they could be cache friendly in theory, but aren't in practice.
    • HideousKojima 1y ago
      Sure, but avoiding CPU cache misses is hardly the only form of optimization. Some optimizations are blatantly obvious. As an example, there was some code I was working on that called the database every single time it was executed. The data it was pulling from the database got updated once a year, and the entire dataset could fit into about 4kb of memory. Caching it in memory was an obvious fix, and given how fast the rest of the code ran once it had this data it was basically a 500% speedup that anyone with half a brain could have roughly estimated.
  • necovek 1y ago
    "Intuitively" literally means without having to learn something. Adding caches or switching to lower-level calls is definitely something learned, and I wouldn't call it "intuitive". What I think you are referring to is that sometimes, simply reading and understanding the code can tell you where the problem really is — still, my experience is that you want to measure the before and after to at least identify the general area you should be looking at, and more often than not, I could figure out what needs optimizing and how without having to get detailed profiles. I did end up being surprised a few times, but it was mostly due to external code like buggy library implementations that didn't do the things they promised (eg. async library really synchronizing everything with a badly placed mutex). At the same time, it's wrong to focus on a single case or a single profile (executing one code path in one set of circumstances), but what you are describing simply sounds like bad engineering — the fact that you can misuse the tool does not make the tool bad, it's still the engineer wielding it who's at fault.