11 ms·

Hotspot performance engineering fails

191 points by slimsag 3y ago

seer 3y ago
One trick I’ve learned is to strive to never be at the limits of the current architecture. Clear requirements are well understood until they aren’t, most projects I’ve been involved in strongly resembled a factorio game - you build a system that’s well designed for its requirements and then the business comes and demands 10 times the size/performance/speed … So I try to have a system where changes and expansions “feel” easy and natural, if they aren’t then thats a big flag to go ahead and start refactoring _now_ so when the business inevitably comes with its changes, you are ready.
samsquire 3y ago
I think if we can make architecture cheap to change, then we can make software that can be drastically optimised from a high level. I think techniques such as differential dataflow, materialized views and scheduling can optimise many performance problems. I often start with something fast and then try add functionality to it to make it useful. Architecture decides your performance characteristics.
mattpallissard 3y ago
Cherry-picking here. > But even if you can find the bottlenecks, they become more numerous with each iteration. Eventually, much of your code needs to be considered. > The effort needed to optimize code grows exponentially. > And that explain why companies do full rewrites of their code for performance: the effort needed to squeeze more performance from the existing code becomes too much and a complete rewrite is cheaper. I agree with this somewhat. Yes the effort needed doesn't scale linearly. Yes you hit a point of diminishing returns. And ABSOLUTELY should you write things with _some_ thought for the future the first time. However, The number of bottlenecks is hardly ever the issue. Most places have a really hard time profiling their applications. They don't know how to effectively. They don't have the instrumentation to do so. They don't think about the time complexity of various operations, standard library methods, storage, and network operations. Rarely does it get more complicated as you knock out the low-hanging fruit. It just gets more tedious. Asking more questions about metrics. Running down more dead-end rabbit holes. And no, there is never an "end"; you just move from one bottleneck to another until you fix whatever your current issue is. This is true for organizations of all shapes, sizes, and skill levels. Taking inefficient code that someone else wrote and fixing it in place has been my shtick for well over a decade now. In my experience a complete re-write, while fun, is almost never cheaper than improving what you already have or incrementally breaking apart your service and re-writing portions of it. If it is you have a small application that doesn't have a lot of man-hours invested in it.
turtleyacht 3y ago
> these lines of code [were] pulling data from memory and software cannot beat Physics. [These] are elementary operations... > measuring big effects is easy, measuring small ones becomes impossible because the action of measuring interacts with the software > to multiply the performance by N, you need ... 2^N optimizations > why companies do full rewrite of their code for performance
- saagarjha 3y ago
  Why quote these lines?
  - turtleyacht 3y ago
    Summarizing the article. Also gives me a way to evaluate performance/optimization. Ideas to hang hooks on.
continuational 3y ago
This is true, when you keep optimizing, you soon face death from a thousand paper cuts. But often, it's enough to find that bottleneck and make it a few times faster.
- hinkley 3y ago
  The solution to this is zone defense instead of man-man. The sad fact is that a manager won't approve you working on something that'll save 1% CPU. But once the tall and medium tent poles have been knocked down, that's all there is left. There are hundreds of them, and they double or triple your response time and/or CPU load. I've had much, much better outcomes by rejecting trying to achieve an N% speedup across the entire app, and instead picking one subject area of the code and finding 20% there. You deep dive into that section, fulling absorbing how it works and why it works, and you fix every problem you see that registers above the noise floor in your perf tool. Some second and third tier performance problems complement each other, and you can avoid one entirely by altering the other. The risk of the 1% changes can be amortized over both the effort you expended learning this code, and the testing time required to validate 3 large changes scattered across the codebase versus 8 changes in the same workflow. Much simpler to explain, much easier to verify. Big wins feel good now but the company comes to expect them. In the place where I used this best, I delivered 20% performance improvements per release for something like 8 releases in a row, before I ran out of areas I hadn't touched before. Often I'd find a perf issue in how the current section of code talks to another, and that would inform what section of code I worked on next, while the problem domain was still fresh in my brain.
sosodev 3y ago
This is true if your end goal is to have a super fast program but that is very rarely the case. The GTA online loading times issues went unnoticed for years because Rockstar just didn't care that the loading times were long. Users still played the game and spent a ton of money. Performance hotspots often are the difference between acceptable and unacceptable performance. I'm sure I'm not the only person who has seen that be the case many times.
- hinkley 3y ago
  I don't think people understand the ways that we have adapted to delays. At least once a month I complain about how when we were kids, commercials were when you went for a pee break or to get a snack. There was no pause button. Bing watching on streaming always means you have to interrupt or wait twenty five minutes. I suspect if you spied on a bunch of GTA players you'd find them launching the game and then going to the fridge, rather than the other way around.
  - usrusr 3y ago
    And it's not impossible at all (even if perhaps not actually likely) that their entire microtransaction business would run noticeably worse if players were able to jump in and out of a session in an instant. That fridge run during launch? It's a sunken cost that should better be worth it. Now they are committed. (edit: reading some other parts it turns out the experiment has actually been made, console versions that never had the startup bug are apparently doing fine)
- eklitzke 3y ago
  >This is true if your end goal is to have a super fast program but that is very rarely the case. This is true in some banal sense, but kind of misses the point that there are certain domains where high performance software is a given, and in other domains it may rarely be important. If you're working on games, certain types of financial systems, autonomous vehicles, operating systems, etc. then high performance is critical and something you need to think about quite literally from day one.
bastawhiz 3y ago
> And that explain why companies do full rewrites of their code for performance: the effort needed to squeeze more performance from the existing code becomes too much and a complete rewrite is cheaper. The article provides reasons why optimization gets harder, but no arguments for why a rewrite is better. It's unclear whether the author is arguing for rewrites or whether they're simply pointing out why companies take them on. Arguably, though, companies taking on a full rewrite surely must have considered the cost of optimization (versus naively saying "the system is slow, replace it!"—though maybe some did). Rewrites are big, expensive, and time-consuming. It means new bugs and unknown unknowns, and no time to add features or fix bugs because you're busy rewriting functional code. It's a scapegoat for lack of improvement or progress. You shouldn't take one on lightly. At the same time, this post also neglects that some efficiency wins have little to do with the efficiency of the code, but rather the efficiency of the logic. An N+1 query in your application looks like your database is slow: you're wasting a ton of time sitting and waiting for your DB to return information! But the real problem is that you're repeatedly going back-and-forth to the database to query lots of little pieces of information that could have far more efficiently been queried all at once. > It is relatively easy to double the performance of an unoptimized piece of code, but much harder to multiply it by 10. You quickly hit walls that can be unsurmountable: the effort needed to double the performance again would just be too much. That's not really true, though. One bad SQL query can go from many seconds or minutes to milliseconds. One accidentally-quadratic algorithm can take orders of magnitude more time than a linear-time algorithm. One bad regexp can account for the majority of a request. Of course, as you fix the biggest performance problems, the only problems left are ones that are smaller than your biggest ones, so you'll have diminishing returns. But it also begs the question, what choices has your existing code made that makes it _ten times_ slower than you want it to be? In my experience, you're doing work synchronously that could have been put in a queue and worked on asynchronously. It's more often "you're doing more work than you should" or "you're being inefficient with the resources you have available" than "a specific piece of code is computationally inefficient".
- nemothekid 3y ago
  >The article provides reasons why optimization gets harder, but no arguments for why a rewrite is better. It's unclear whether the author is arguing for rewrites or whether they're simply pointing out why companies take them on. He didn't argue a rewrite is just "better"; his argument was that a rewrite was the only card of the table. The architecture was deficient and to get more performance you have to change the architecture, which means a rewrite. I tend to agree; I take the view that most engineers are smart, and compilers/interpreters/virtual machines are even smarter so most targeted optimizations aren't going to result in very much gain. A codebase full of N+1 queries or unindexed queries never cared about performance to begin with. For true gains, you will have to think about data which is the true bottleneck for most applications - getting data from memory, the disk or the network will be much longer that any instruction cycle. The way memory moves through your application is baked into your architecture and changing this will almost always involve a rewrite. To your final point, >In my experience, you're doing work synchronously that could have been put in a queue and worked on asynchronously. moving from a synchronous codebase to an async one almost always involves a rewrite.
ummonk 3y ago
From my experience it's better to just consider performance from the get-go, and carefully consider which tech stack you're using and how the specific logic / system architecture you've chosen will be performant. It's much easier than being stuck with performance problems down the road that will need a painful rewrite. The whole mantra of avoiding "premature optimizations" was applicable in an era when "optimizations" meant rewriting C code in assembly.
- gashmol 3y ago
  IMHO the crutial word is not premature but rather optimaization. Not everything is an optimazation. Merging two loops into one is an optimazion that should be postponed until it's necessary and the system have been profiled. Joining two large lists by a hash join instead of a nested loop is a design decision that should be made at the outset.
- smolder 3y ago
  The other thing that's changed from the 'every optimization is premature' era is that shrinking CPUs don't result in big gains in frequency anymore -- Moore's law isn't going to make your python run at C speed no matter how long you wait for better hardware.
  - 0x000xca0xfe 3y ago
    And the speed of light ensures that memory latencies won't get much better until CPUs are small cubes made of SRAM.
    - actionfromafar 3y ago
      Ehrm. Small spheres of SRAM, if I may.
      - speed_spread 3y ago
        That's one way of dealing with corner cases.
PaulHoule 3y ago
I went through a time when I was pitching a "boxes-and-lines" data processing tool like https://www.knime.com/ https://www.knime.com/ which more-or-less passed JSON documents (instead of SQL rows) over the lines and found that the kind of people who bought and finance database startups wouldn't touch anything that couldn't be implemented with columnar processing. I thought that this kind of system would advance the "low code" nature of these systems because with relational rows many kinds of data processing require splitting up the data into streams and joining them whereas an object-relational system lets you localize processing in a small area of the graph and also be able to reuse parts of a computation. Columnar processing is so much faster than row-based processing and most investors and partners thought that customers really needed speed at the expense of being able to write simpler pipelines. Even though I had a nice demo of a hybrid batch/stream processing system (that gave correct answers), none of them cared. Thus, from one viewpoint, architecture is everything. (Funny though, I later worked for a company that had a system like this that wasn't quite sure what algebra the tool worked on and the tool didn't quite always get the same answer on each run...)
MattPalmer1086 3y ago
It was interesting to read about why hotspots are not the whole story in performance. They are still important though. Facebook may have the resources and/or need to do complete rewrites of everything to squeeze out more performance, but most companies don't. I've personally improved performance of a lot of code significantly by identifying hot spots. So calling hotspot performance engineering a fail seems a bit unnecessarily provocative.
- jesse__ 3y ago
  > Facebook may have the resources and/or need to do complete rewrites of everything to squeeze out more performance, but most companies don't. Actually, if you watch the video Casey put together, he very clearly demonstrates most companies _do_.
  - MattPalmer1086 3y ago
    Hardly. He takes evidence from Facebook, Twitter and Uber. These are not small companies and they operate at vast scale, unlike the majority of companies who engage in software development.
0x000xca0xfe 3y ago
Optimizing for modern CPUs means optimizing for predictable memory accesses and program flow. Minimizing memory usage helps a lot, too. Unfortunately this is pretty counterintuitive and most programming languages do not make it easy. And if you optimize for size you almost get laughed at.
- usefulcat 3y ago
  > And if you optimize for size you almost get laughed at. Only by people who don't know much about optimization, which is the real problem. Memory caches are obviously of fixed size and relatively small.
helen___keller 3y ago
This whole thing is basically a straw man. “Performance engineering works but sometimes it’s not enough to overcome a bad architecture”. Alright, was that actually in question in the first place?
- vgatherps 3y ago
  You'd be surprised how common the view "Performance doesn't matter now we'll just fix the hotspot later" is
  - helen___keller 3y ago
    Again, that’s a straw man. All of engineering is a trade off. You trade off performance to complete the implementation faster, you keep in mind the amount of effort you are trading off in the future in the chance you are wrong and this will need to be optimized. That’s not the same as “performance doesn’t matter”. People who say “performance doesn’t matter we can optimize later” are using this as a shorthand for, time now is significantly more valuable than time in the future. For example, we must hit a deadline or must get a POC out.
tasubotadas 3y ago
The guy invents a strawman to justify premature optimization.
charcircuit 3y ago
It's always about architecture. In the micro these are the hotspots you optimize in the macro these are the large rewrites you see. Performance is not the only thing that you should optimize your architecture for. Factors like adaptability, robustness, ease of understanding, speed of implementation, maintance cost, etc are things that you should consider. The factors that are the best today are not always still the best in the future which is why rewrites are a part of any software's life cycle.
- aranchelk 3y ago
  That stuff is boring. Don’t be a killjoy.
manv1 3y ago
TL; DR: "It's better to design a fast system from the get-go instead of trying to fix a slow system later." That's basically true. I worked on a system that was Java/scala/spring/hibernate and it was just slow. It was slow when it was servicing an empty request, and it just went downhill from there. They just built it wrong...and they went ahead and built it wrong again. Today, I could replace it was a few hundred lines of node in AWS/Lambda and get multiple orders of magnitude of performance.
- tonyarkles 3y ago
  > Today, I could replace it was a few hundred lines of node in AWS/Lambda and get multiple orders of magnitude of performance. I had a fun bake-off a few years back. I was in more of a devOPS role (i.e. mostly Ops but writing code here and there when needed) and we needed something akin to an API Gateway but with some very domain-specific routing logic. One of the developers and I talked it through, he wanted to do Node, I suggested it would be a perfect place for Go. We decided to do two parallel (~500 LOC) implementations over a weekend and run them head-to-head on Monday. The code, logically, ended up coming out quite similar, which made us both pretty happy. Then... we started the benchmarking. They were neck and neck! For a fixed level of throughput, Go was only winning by maybe 5% on latency. That stayed true up until about 10krps, at which point Node flatlined because it was saturating a single CPU and Go just kept going and going and going until it saturated all of the cores on the VM we were testing on. Could we have scaled out the Node version to multiple nodes in the cluster? Sure. At 10krps though, it was already using 2-3x the RAM that the Go version was using at 80krps, and replicating 8 copies of it vs the 2x we did with the Go version (just for redundancy) starts to have non-trivial resource costs. And don't get me wrong, we had a bunch of the exact same Java/scala/spring/hibernate type stuff in the system as well, and it was dog-ass slow in comparison while also eating RAM like it was candy.
  - manv1 3y ago
    Yeah, the one time I used go it was pretty good. The big question is always whether your stuff spends more time waiting or more time processing. For the former, it's node. For the latter, it's go.
NovemberWhiskey 3y ago
I don't think I entirely agree with the premise here. Yes, it is extremely difficult to engineer performance in after the fact; but assuming you've got an architecture that's basically fit for purpose (from the performance perspective), then improving by targeting hotspots is sound, isn't it? That's literally Amdahl's law.
- Sesse__ 3y ago
  Amdahl’s law is specifically about the futility of optimizing by removing hotspots... (Or rather, that it can only take you so far.)
  - taneq 3y ago
    No, it's about the limitations of optimizing individual parts of a program. A corollary would be the futility of optimizing non-hotspots if you want any significant gain. If your code spends 80% of its time in a given function you can't speed up the overall task by more than 500% by optimizing that function alone. If it spends 1% of its time in a given function then you can't speed it up by more than 1.1% by optimizing that function.
    - astrange 3y ago
      Performance isn't all first order wallclock time stuff. If you rarely enter one part of the program but that part accidentally deletes all your caches, everything else is going to get slower afterward.
- taneq 3y ago
  Yeah, I'm with you. The 80/20 rule isn't about "write code slow then speed up the inner loops", it's about "don't waste time optimizing the slow bits, which means don't waste time optimizing anything until you know which bits are the slow bits." It's explicitly about micro-optimizations, not architecture. And it's about helping devs avoid bogging down doing pointless work making micro-optimizations that will never affect the performance metrics that you care about.
titzer 3y ago
Just optimizing the "hotspots" is not the only strategy to improving performance. As Daniel alludes to towards the end, it's about maximizing effort versus payoff. E.g. you might find a hotspot that indicates 20% of the time is spent in a very small function. Yet that small function might be already close to optimal. After intense effort, you improve this function by 25%. Great, you saved 5% overall. But there might be a 5% somewhere else in the program that shouldn't exist at all, and if you remove it, you also get a 5% speedup. Another example is that maybe you have a loop around 10 function calls that calls each of them 5 times. Each of them might show up as 10% of the execution time. But then you discover that the entire loop only needs to execute once. So you can speed up the program by 5x by only iterating once. Yet the proportion in the profile doesn't change; the program just does 5x less work. Profiles can be deceiving.
nxpnsv 3y ago
I certainly have had several great successes with profiling, i wonder what i have been doing wrong….
- usefulcat 3y ago
  I don't think he's saying "don't profile", more like "don't assume that all programs spend 80% of the time in 20% of the code". I know I've seen plenty of cases where the function at the top of the callgrind output is ~5% or less of total execution time.
  - nxpnsv 3y ago
    Valid point
  - rightbyte 3y ago
    Ye. I mean increasing the performance of semi-perpormant apps is really hard. It's tiny changes everywhere. At some point you got even the high hanging fruit and you are changing the order of if-clauses to decrease cache footprint to increase speed.
ilrwbwrkhv 3y ago
As an aside I am so grateful for people like Casey Muratori (whose article is referenced in that blog). Their rants about speed and performance is so needed in this world where a chat app like Slack literally gets the fans spinning. There are few of them remaining but they shine brightly in an otherwise messy and slow land. Recently I had to touch a nuxt js project and my experience with it was so janky. The hot module replacement sometimes works, sometimes doesn't and basically it is worse than having to refresh manually every single time.
kens 3y ago
Knuth's comments on optimization are very nuanced. I beg people to not use the phrase "premature optimization" without reading what he actually said. See page 8 of https://pic.plover.com/knuth-GOTO.pdf https://pic.plover.com/knuth-GOTO.pdf
kristjansson 3y ago
Hunting hotspots is a great way to find out where you’ve done dumb things, but it can’t show you where you haven’t done smart things.
dang 3y ago
Recent and related: Performance excuses debunked - https://news.ycombinator.com/item?id=35718912 https://news.ycombinator.com/item?id=35718912 - April 2023 (118 comments)

191 points by slimsag 3y ago

seer 3y ago

One trick I’ve learned is to strive to never be at the limits of the current architecture. Clear requirements are well understood until they aren’t, most projects I’ve been involved in strongly resembled a factorio game - you build a system that’s well designed for its requirements and then the business comes and demands 10 times the size/performance/speed … So I try to have a system where changes and expansions “feel” easy and natural, if they aren’t then thats a big flag to go ahead and start refactoring _now_ so when the business inevitably comes with its changes, you are ready.

samsquire 3y ago

I think if we can make architecture cheap to change, then we can make software that can be drastically optimised from a high level. I think techniques such as differential dataflow, materialized views and scheduling can optimise many performance problems. I often start with something fast and then try add functionality to it to make it useful. Architecture decides your performance characteristics.

mattpallissard 3y ago

Cherry-picking here. > But even if you can find the bottlenecks, they become more numerous with each iteration. Eventually, much of your code needs to be considered. > The effort needed to optimize code grows exponentially. > And that explain why companies do full rewrites of their code for performance: the effort needed to squeeze more performance from the existing code becomes too much and a complete rewrite is cheaper. I agree with this somewhat. Yes the effort needed doesn't scale linearly. Yes you hit a point of diminishing returns. And ABSOLUTELY should you write things with _some_ thought for the future the first time. However, The number of bottlenecks is hardly ever the issue. Most places have a really hard time profiling their applications. They don't know how to effectively. They don't have the instrumentation to do so. They don't think about the time complexity of various operations, standard library methods, storage, and network operations. Rarely does it get more complicated as you knock out the low-hanging fruit. It just gets more tedious. Asking more questions about metrics. Running down more dead-end rabbit holes. And no, there is never an "end"; you just move from one bottleneck to another until you fix whatever your current issue is. This is true for organizations of all shapes, sizes, and skill levels. Taking inefficient code that someone else wrote and fixing it in place has been my shtick for well over a decade now. In my experience a complete re-write, while fun, is almost never cheaper than improving what you already have or incrementally breaking apart your service and re-writing portions of it. If it is you have a small application that doesn't have a lot of man-hours invested in it.

turtleyacht 3y ago

> these lines of code [were] pulling data from memory and software cannot beat Physics. [These] are elementary operations... > measuring big effects is easy, measuring small ones becomes impossible because the action of measuring interacts with the software > to multiply the performance by N, you need ... 2^N optimizations > why companies do full rewrite of their code for performance

saagarjha 3y ago
Why quote these lines?
- turtleyacht 3y ago
  Summarizing the article. Also gives me a way to evaluate performance/optimization. Ideas to hang hooks on.

continuational 3y ago

This is true, when you keep optimizing, you soon face death from a thousand paper cuts. But often, it's enough to find that bottleneck and make it a few times faster.

hinkley 3y ago
The solution to this is zone defense instead of man-man. The sad fact is that a manager won't approve you working on something that'll save 1% CPU. But once the tall and medium tent poles have been knocked down, that's all there is left. There are hundreds of them, and they double or triple your response time and/or CPU load. I've had much, much better outcomes by rejecting trying to achieve an N% speedup across the entire app, and instead picking one subject area of the code and finding 20% there. You deep dive into that section, fulling absorbing how it works and why it works, and you fix every problem you see that registers above the noise floor in your perf tool. Some second and third tier performance problems complement each other, and you can avoid one entirely by altering the other. The risk of the 1% changes can be amortized over both the effort you expended learning this code, and the testing time required to validate 3 large changes scattered across the codebase versus 8 changes in the same workflow. Much simpler to explain, much easier to verify. Big wins feel good now but the company comes to expect them. In the place where I used this best, I delivered 20% performance improvements per release for something like 8 releases in a row, before I ran out of areas I hadn't touched before. Often I'd find a perf issue in how the current section of code talks to another, and that would inform what section of code I worked on next, while the problem domain was still fresh in my brain.

sosodev 3y ago

This is true if your end goal is to have a super fast program but that is very rarely the case. The GTA online loading times issues went unnoticed for years because Rockstar just didn't care that the loading times were long. Users still played the game and spent a ton of money. Performance hotspots often are the difference between acceptable and unacceptable performance. I'm sure I'm not the only person who has seen that be the case many times.

hinkley 3y ago
I don't think people understand the ways that we have adapted to delays. At least once a month I complain about how when we were kids, commercials were when you went for a pee break or to get a snack. There was no pause button. Bing watching on streaming always means you have to interrupt or wait twenty five minutes. I suspect if you spied on a bunch of GTA players you'd find them launching the game and then going to the fridge, rather than the other way around.
- usrusr 3y ago
  And it's not impossible at all (even if perhaps not actually likely) that their entire microtransaction business would run noticeably worse if players were able to jump in and out of a session in an instant. That fridge run during launch? It's a sunken cost that should better be worth it. Now they are committed. (edit: reading some other parts it turns out the experiment has actually been made, console versions that never had the startup bug are apparently doing fine)
eklitzke 3y ago
>This is true if your end goal is to have a super fast program but that is very rarely the case. This is true in some banal sense, but kind of misses the point that there are certain domains where high performance software is a given, and in other domains it may rarely be important. If you're working on games, certain types of financial systems, autonomous vehicles, operating systems, etc. then high performance is critical and something you need to think about quite literally from day one.

bastawhiz 3y ago

> And that explain why companies do full rewrites of their code for performance: the effort needed to squeeze more performance from the existing code becomes too much and a complete rewrite is cheaper. The article provides reasons why optimization gets harder, but no arguments for why a rewrite is better. It's unclear whether the author is arguing for rewrites or whether they're simply pointing out why companies take them on. Arguably, though, companies taking on a full rewrite surely must have considered the cost of optimization (versus naively saying "the system is slow, replace it!"—though maybe some did). Rewrites are big, expensive, and time-consuming. It means new bugs and unknown unknowns, and no time to add features or fix bugs because you're busy rewriting functional code. It's a scapegoat for lack of improvement or progress. You shouldn't take one on lightly. At the same time, this post also neglects that some efficiency wins have little to do with the efficiency of the code, but rather the efficiency of the logic. An N+1 query in your application looks like your database is slow: you're wasting a ton of time sitting and waiting for your DB to return information! But the real problem is that you're repeatedly going back-and-forth to the database to query lots of little pieces of information that could have far more efficiently been queried all at once. > It is relatively easy to double the performance of an unoptimized piece of code, but much harder to multiply it by 10. You quickly hit walls that can be unsurmountable: the effort needed to double the performance again would just be too much. That's not really true, though. One bad SQL query can go from many seconds or minutes to milliseconds. One accidentally-quadratic algorithm can take orders of magnitude more time than a linear-time algorithm. One bad regexp can account for the majority of a request. Of course, as you fix the biggest performance problems, the only problems left are ones that are smaller than your biggest ones, so you'll have diminishing returns. But it also begs the question, what choices has your existing code made that makes it _ten times_ slower than you want it to be? In my experience, you're doing work synchronously that could have been put in a queue and worked on asynchronously. It's more often "you're doing more work than you should" or "you're being inefficient with the resources you have available" than "a specific piece of code is computationally inefficient".

nemothekid 3y ago
>The article provides reasons why optimization gets harder, but no arguments for why a rewrite is better. It's unclear whether the author is arguing for rewrites or whether they're simply pointing out why companies take them on. He didn't argue a rewrite is just "better"; his argument was that a rewrite was the only card of the table. The architecture was deficient and to get more performance you have to change the architecture, which means a rewrite. I tend to agree; I take the view that most engineers are smart, and compilers/interpreters/virtual machines are even smarter so most targeted optimizations aren't going to result in very much gain. A codebase full of N+1 queries or unindexed queries never cared about performance to begin with. For true gains, you will have to think about data which is the true bottleneck for most applications - getting data from memory, the disk or the network will be much longer that any instruction cycle. The way memory moves through your application is baked into your architecture and changing this will almost always involve a rewrite. To your final point, >In my experience, you're doing work synchronously that could have been put in a queue and worked on asynchronously. moving from a synchronous codebase to an async one almost always involves a rewrite.

ummonk 3y ago

From my experience it's better to just consider performance from the get-go, and carefully consider which tech stack you're using and how the specific logic / system architecture you've chosen will be performant. It's much easier than being stuck with performance problems down the road that will need a painful rewrite. The whole mantra of avoiding "premature optimizations" was applicable in an era when "optimizations" meant rewriting C code in assembly.

gashmol 3y ago
IMHO the crutial word is not premature but rather optimaization. Not everything is an optimazation. Merging two loops into one is an optimazion that should be postponed until it's necessary and the system have been profiled. Joining two large lists by a hash join instead of a nested loop is a design decision that should be made at the outset.
smolder 3y ago
The other thing that's changed from the 'every optimization is premature' era is that shrinking CPUs don't result in big gains in frequency anymore -- Moore's law isn't going to make your python run at C speed no matter how long you wait for better hardware.
- 0x000xca0xfe 3y ago
  And the speed of light ensures that memory latencies won't get much better until CPUs are small cubes made of SRAM.
  - actionfromafar 3y ago
    Ehrm. Small spheres of SRAM, if I may.
    - speed_spread 3y ago
      That's one way of dealing with corner cases.

PaulHoule 3y ago

I went through a time when I was pitching a "boxes-and-lines" data processing tool like https://www.knime.com/ https://www.knime.com/ which more-or-less passed JSON documents (instead of SQL rows) over the lines and found that the kind of people who bought and finance database startups wouldn't touch anything that couldn't be implemented with columnar processing. I thought that this kind of system would advance the "low code" nature of these systems because with relational rows many kinds of data processing require splitting up the data into streams and joining them whereas an object-relational system lets you localize processing in a small area of the graph and also be able to reuse parts of a computation. Columnar processing is so much faster than row-based processing and most investors and partners thought that customers really needed speed at the expense of being able to write simpler pipelines. Even though I had a nice demo of a hybrid batch/stream processing system (that gave correct answers), none of them cared. Thus, from one viewpoint, architecture is everything. (Funny though, I later worked for a company that had a system like this that wasn't quite sure what algebra the tool worked on and the tool didn't quite always get the same answer on each run...)

MattPalmer1086 3y ago

It was interesting to read about why hotspots are not the whole story in performance. They are still important though. Facebook may have the resources and/or need to do complete rewrites of everything to squeeze out more performance, but most companies don't. I've personally improved performance of a lot of code significantly by identifying hot spots. So calling hotspot performance engineering a fail seems a bit unnecessarily provocative.

jesse__ 3y ago
> Facebook may have the resources and/or need to do complete rewrites of everything to squeeze out more performance, but most companies don't. Actually, if you watch the video Casey put together, he very clearly demonstrates most companies _do_.
- MattPalmer1086 3y ago
  Hardly. He takes evidence from Facebook, Twitter and Uber. These are not small companies and they operate at vast scale, unlike the majority of companies who engage in software development.

0x000xca0xfe 3y ago

Optimizing for modern CPUs means optimizing for predictable memory accesses and program flow. Minimizing memory usage helps a lot, too. Unfortunately this is pretty counterintuitive and most programming languages do not make it easy. And if you optimize for size you almost get laughed at.

usefulcat 3y ago
> And if you optimize for size you almost get laughed at. Only by people who don't know much about optimization, which is the real problem. Memory caches are obviously of fixed size and relatively small.

helen___keller 3y ago

This whole thing is basically a straw man. “Performance engineering works but sometimes it’s not enough to overcome a bad architecture”. Alright, was that actually in question in the first place?

vgatherps 3y ago
You'd be surprised how common the view "Performance doesn't matter now we'll just fix the hotspot later" is
- helen___keller 3y ago
  Again, that’s a straw man. All of engineering is a trade off. You trade off performance to complete the implementation faster, you keep in mind the amount of effort you are trading off in the future in the chance you are wrong and this will need to be optimized. That’s not the same as “performance doesn’t matter”. People who say “performance doesn’t matter we can optimize later” are using this as a shorthand for, time now is significantly more valuable than time in the future. For example, we must hit a deadline or must get a POC out.

tasubotadas 3y ago

The guy invents a strawman to justify premature optimization.

charcircuit 3y ago

It's always about architecture. In the micro these are the hotspots you optimize in the macro these are the large rewrites you see. Performance is not the only thing that you should optimize your architecture for. Factors like adaptability, robustness, ease of understanding, speed of implementation, maintance cost, etc are things that you should consider. The factors that are the best today are not always still the best in the future which is why rewrites are a part of any software's life cycle.

aranchelk 3y ago
That stuff is boring. Don’t be a killjoy.

manv1 3y ago

TL; DR: "It's better to design a fast system from the get-go instead of trying to fix a slow system later." That's basically true. I worked on a system that was Java/scala/spring/hibernate and it was just slow. It was slow when it was servicing an empty request, and it just went downhill from there. They just built it wrong...and they went ahead and built it wrong again. Today, I could replace it was a few hundred lines of node in AWS/Lambda and get multiple orders of magnitude of performance.

tonyarkles 3y ago
> Today, I could replace it was a few hundred lines of node in AWS/Lambda and get multiple orders of magnitude of performance. I had a fun bake-off a few years back. I was in more of a devOPS role (i.e. mostly Ops but writing code here and there when needed) and we needed something akin to an API Gateway but with some very domain-specific routing logic. One of the developers and I talked it through, he wanted to do Node, I suggested it would be a perfect place for Go. We decided to do two parallel (~500 LOC) implementations over a weekend and run them head-to-head on Monday. The code, logically, ended up coming out quite similar, which made us both pretty happy. Then... we started the benchmarking. They were neck and neck! For a fixed level of throughput, Go was only winning by maybe 5% on latency. That stayed true up until about 10krps, at which point Node flatlined because it was saturating a single CPU and Go just kept going and going and going until it saturated all of the cores on the VM we were testing on. Could we have scaled out the Node version to multiple nodes in the cluster? Sure. At 10krps though, it was already using 2-3x the RAM that the Go version was using at 80krps, and replicating 8 copies of it vs the 2x we did with the Go version (just for redundancy) starts to have non-trivial resource costs. And don't get me wrong, we had a bunch of the exact same Java/scala/spring/hibernate type stuff in the system as well, and it was dog-ass slow in comparison while also eating RAM like it was candy.
- manv1 3y ago
  Yeah, the one time I used go it was pretty good. The big question is always whether your stuff spends more time waiting or more time processing. For the former, it's node. For the latter, it's go.

NovemberWhiskey 3y ago

I don't think I entirely agree with the premise here. Yes, it is extremely difficult to engineer performance in after the fact; but assuming you've got an architecture that's basically fit for purpose (from the performance perspective), then improving by targeting hotspots is sound, isn't it? That's literally Amdahl's law.

Sesse__ 3y ago
Amdahl’s law is specifically about the futility of optimizing by removing hotspots... (Or rather, that it can only take you so far.)
- taneq 3y ago
  No, it's about the limitations of optimizing individual parts of a program. A corollary would be the futility of optimizing non-hotspots if you want any significant gain. If your code spends 80% of its time in a given function you can't speed up the overall task by more than 500% by optimizing that function alone. If it spends 1% of its time in a given function then you can't speed it up by more than 1.1% by optimizing that function.
  - astrange 3y ago
    Performance isn't all first order wallclock time stuff. If you rarely enter one part of the program but that part accidentally deletes all your caches, everything else is going to get slower afterward.
taneq 3y ago
Yeah, I'm with you. The 80/20 rule isn't about "write code slow then speed up the inner loops", it's about "don't waste time optimizing the slow bits, which means don't waste time optimizing anything until you know which bits are the slow bits." It's explicitly about micro-optimizations, not architecture. And it's about helping devs avoid bogging down doing pointless work making micro-optimizations that will never affect the performance metrics that you care about.

titzer 3y ago

Just optimizing the "hotspots" is not the only strategy to improving performance. As Daniel alludes to towards the end, it's about maximizing effort versus payoff. E.g. you might find a hotspot that indicates 20% of the time is spent in a very small function. Yet that small function might be already close to optimal. After intense effort, you improve this function by 25%. Great, you saved 5% overall. But there might be a 5% somewhere else in the program that shouldn't exist at all, and if you remove it, you also get a 5% speedup. Another example is that maybe you have a loop around 10 function calls that calls each of them 5 times. Each of them might show up as 10% of the execution time. But then you discover that the entire loop only needs to execute once. So you can speed up the program by 5x by only iterating once. Yet the proportion in the profile doesn't change; the program just does 5x less work. Profiles can be deceiving.

nxpnsv 3y ago

I certainly have had several great successes with profiling, i wonder what i have been doing wrong….

usefulcat 3y ago
I don't think he's saying "don't profile", more like "don't assume that all programs spend 80% of the time in 20% of the code". I know I've seen plenty of cases where the function at the top of the callgrind output is ~5% or less of total execution time.
- nxpnsv 3y ago
  Valid point
- rightbyte 3y ago
  Ye. I mean increasing the performance of semi-perpormant apps is really hard. It's tiny changes everywhere. At some point you got even the high hanging fruit and you are changing the order of if-clauses to decrease cache footprint to increase speed.

ilrwbwrkhv 3y ago

As an aside I am so grateful for people like Casey Muratori (whose article is referenced in that blog). Their rants about speed and performance is so needed in this world where a chat app like Slack literally gets the fans spinning. There are few of them remaining but they shine brightly in an otherwise messy and slow land. Recently I had to touch a nuxt js project and my experience with it was so janky. The hot module replacement sometimes works, sometimes doesn't and basically it is worse than having to refresh manually every single time.

kens 3y ago

Knuth's comments on optimization are very nuanced. I beg people to not use the phrase "premature optimization" without reading what he actually said. See page 8 of https://pic.plover.com/knuth-GOTO.pdf https://pic.plover.com/knuth-GOTO.pdf

kristjansson 3y ago

Hunting hotspots is a great way to find out where you’ve done dumb things, but it can’t show you where you haven’t done smart things.

dang 3y ago

Recent and related: Performance excuses debunked - https://news.ycombinator.com/item?id=35718912 https://news.ycombinator.com/item?id=35718912 - April 2023 (118 comments)