12 ms·

> But if performance work has been neglected for most of the product development cycle, chances are that when you finally fire up the profiler it’ll be a unifor

by blt 8y ago

> But if performance work has been neglected for most of the product development cycle, chances are that when you finally fire up the profiler it’ll be a uniformly slow mess with deep systemic issues. Totally true, and I have observed it IRL on large, old projects whose architecture was hostile to performance. And these products were doing computationally intense stuff. > Performance is everyone’s responsibility and it needs to be part of the process along the way. Yes. Everyone repeats "only optimize after profiling". It's true: if your program is running too slowly, you should always profile. But that rule only applies when your code is already slow. It doesn't actually say anything about the rest of the development process. Developers should have a solid grasp of computer hardware, compilers, interpreters, operating systems, and architecture of high performance software. Obliviousness to these is the source of performance-hostile program designs. If you know these things, you will write better performing code without consciously thinking about it. If this knowledge is weighed more heavily in the software dev community, people will put in the effort to learn it. It's not that complicated. If the average dev replaced half of their effort learning languages, libraries, and design patterns with effort learning these fundamentals, the field would be in a much better place.

mlthoughts2018 8y ago
My experience writing large scale search engine, NLP and computer vision services, generally as backend web services that have tight performance constraints for other client apps that consume from them, is basically fundamentally the opposite of what you say and what you quoted from the article. It is virtually never the case that profiling reveals a uniformly slow mess of code, and in many cases this doesn’t even make sense as a possibility because you write modular performance tests and profiling tools the same as you write modular unit tests. You would never fire up the profiler and just naively profile a whole backend service all at once, apart from maybe getting extremely coarse latency statistics that only matter in terms of what stakeholder constraints you have to meet. For analyzing cases when you’re not just gathering whole-service stats for someone else, you’d always profile in a more granular, experiment-driven way. > “Developers should have a solid grasp of computer hardware, compilers, interpreters, operating systems, and architecture of high performance software. Obliviousness to these is the source of performance-hostile program designs. If you know these things, you will write better performing code without consciously thinking about it.” While it’s good to know more about more things generally, I think the specific claim that any of this will help you design better code in the first place is totally and emphatically wrong. Instead it leads you down wrong tracks of wasted effort where someone’s “experience and intuition” about necessary optimizations or necessary architectural trade-offs ends up being irrelevant to the eventual end goal, the changing requirements from the sales team, etc. etc. This happens so frequently and clashes so miserably with a basic YAGNI pragmatism that it really is a good heuristic to just say that early focus on optimization characteristics is always premature. In the most optimization-critical applications I’ve worked on, getting primitive and poorly performing prototypes up and running for stakeholders has always been the most critical part to completing the project within the necessary performance constraints. Treating it like an iterative, empirical problem to solve with profiling is absolutely a sacred virtue for pragmatically solving the problem, and carries far, far less risk than precommitting to choices made on the speculative basis of performance characteristics borne out of someone else’s “experience” with performance-intensive concepts or techniques. Such experience really only matters later on, long after you’ve gathered evidence from profiling.
- hinkley 8y ago
  The performance antagonistic design they’re talking about can’t benefit from what you’re talking about. Worse, it’s usually antagonistic to other things like well-contained modifications. For instance, look at the Big Ball of Mud design. When you are looking at a Big Ball of Mud, these per module performance tests are difficult or impossible to maintain. It’s quite common to see a flame chart with high entropy. Nothing “stands out” after the second or third round of tweaking. Yes there are ways to proceed at this point, but they don’t appear in the literature
- gridspy 8y ago
  The quote you mention is specifically talking about the architecture / design of the optimal program. Once a design such as a linked-list of polymorphic classes being iterated on in a loop is established, it is hard to refactor. Those with performance experience would understand that this design will have caching, branch prediction and similar issues. However, I totally agree with you that the overall implementation should be driven by iterative coding and profiling. BUT when it is time to choose memory layouts, data models, etc - planning in advance with an eye to performance is very valuable.
- dman 8y ago
  I think the issue here is that the large scale systems you cite usually have performance being dominated by the relatively high cost of making a network hop. They dominate runtime to such an extent that unless you are writing code with which is >= O(n^2), your runtime is almost entirely dominated by network latency and the number of hops you make. Also many problems in these domains are embarrasingly parallel and you can effectively summon large number of machines to hold state for you in memory. Where it does get interesting is situations like games / garbage collectors where you want to minimise latency. On such problems you usually cant bolt on performance after the fact.
  - mlthoughts2018 8y ago
    No, this is not at all what I’m talking about. I’m specifically talking about cases with tight latency requirements after the request has been received. Like receiving a POST request containing many images, and invoking complex image preprocessing with a mix of e.g. opencv and homemade algorithms and then feeding into a deep convolutional neural network. Latencies of 2-3 seconds per request might be the target, and a naive unoptimized implementation of just the backend code (totally unrelated to any network parts) might result in something 10x too slow. In that case, it’s absolutely a best practice to start out with that 10x-too-slow-but-easy-to-implement-for-stakeholders version, and only harden or tighten your homemade computer vision code or use of opencv or some neural network tool after profiling. > “Where it does get interesting is situations like games / garbage collectors where you want to minimise latency. On such problems you usually cant bolt on performance after the fact.” No, that’s exactly the sort of case I’m talking about. It’s not only possibly to refactor for performance after the fact, but dramatically easier to do so, less risky in terms of optimizing for the wrong things, and less prone to re-work.
hinkley 8y ago
We need a name for this. The old advice is to ignore performance becaus it’s not important. When it becomes important there is now very little you can do. You get the low hanging fruit (terrible analogy. Ask a fruit grower how stupid this is), or the tall tent poles. Eventually you have an unending field of nearly identically tall poles, everyone claims victory (we’ve done all we can!). But your sales team is still mad because your product is several times slower than the competing products. What do we call this? Picket fence? Blanket of doom? Death shroud? Forest for the trees? Stupidity?
- IncRnd 8y ago
  > The old advice is to ignore performance because it’s not important. I think you misunderstood the instructions and anecdotes that you were given about how and when to optimize. Alternatively, your definition of "old" in "old advice" is a very short time in the past.
  - hinkley 8y ago
    The only yardstick of teaching that means anything is to look at what people go and do with what you taught them. I spent a big chunk of my early career doing performance analysis work in a language stereotyped as being slow. My thesis was that it was down to developer skill and not the language. By the time someone spoke Knuth's aphorism at me it was people who were standing between me and the strategic goals of the company. Using it as a shield to get out of doing the hard work to match a competitor or customer expectations. As I quickly discovered, there are a whole set of performance optimizations that also improve readability, and that got around the friction with the foot dragging Knuth quoters. There's a host of other things never show up in the literature. It's a shame, really.
    - basilgohar 8y ago
      A lot of this is due to misunderstanding the quote and knowing the bounds where it applies. Premature optimization being the root of all evil means those misapplying the addage are using it as a crutch. Good code is both performant and also easy to scale because of how it's designed (as mentioned elsewhere in this thread). See [0]. It is suspect that most folks ever got it to "Make it right" in the first place. Executive pressure and lack of support from managers needs to be called out as well as contributing to problems with quality of code ad teams oftentimes are not given the chance to make it right under pressure to deliver under unrealistic deadlines. [0] http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast
brightball 8y ago
Honestly, I think you're setting too high of a bar. If developers learn 2 basic things: 1. How SQL queries work 2. Network latency exists. Moving stuff between your database and your server can get expensive. You'll solve _most_ of the issues that I've seen in my career to this point. There are a few big ones that came from abusing API's too, but it's really the same root issue: You're either thinking about how much information is being moved around or you're not.
- lomnakkus 8y ago
  > You're either thinking about how much information is being moved around or you're not. Not to put too fine a point on it, but I think you meant how often information is being moved around. That's the scenario where latency has a disproportionate effect on performance. (Bandwidth is usually not a problem these days unless you're doing something at the very high end.)
- somenewacc 8y ago
  For CPU bound stuff, the 80%/20% rule is to learn that an array/vector is typically much faster than a linked list, and this is due to how CPU caches work. (this is thinking about how much information is being moved around, but between the processor and RAM) Also, for parallel stuff: locks tend to be bottlenecks. Writing parallel code is tricky overall. Stick to libraries that provide high level lockless data structures like mpsc queues if possible.
  - adrianN 8y ago
    Who uses linked lists? Outside of niche algorithms that splice lots of lists together there are practically zero occasions where linked lists beat dynamic arrays.
    - deleted 8y ago
      [deleted]
    - hvidgaard 8y ago
      Linked lists are very common, especially for non performance critical code. Dynamic arrays either carry a large performance cost at expansion, or is rather complicated to implement in a amortized way. Linked list in comparison is simple, and since it doesn't require a large continues block of memory, the memory pressure is smaller.
didibus 8y ago
I thought it was don't optimize the code prematurely, but at the architechture level, scale and performance must definitly be defined at design time.
MaxBarraclough 8y ago
You've made no mention of what strikes me as a pretty central requirement: at least a basic understanding of data-structures and algorithms. It's a plus to have an understanding of advanced topics like the workings of compilers, but when the foundations are lacking, you're shot. When developers lack a basic grasp of basic data-structures, it harms not only performance, but complexity, readability, maintainability, code-density, and just about everything else we care about.
HelloNurse 8y ago
> Everyone repeats "only optimize after profiling". It's true: if your program is running too slowly, you should always profile. But that rule only applies when your code is already slow. It doesn't actually say anything about the rest of the development process. Except for unusually throwaway or easy to make fast enough projects, software should also be profiled when it runs fast, before there are performance problems. Profiling the application from the beginning ensures readiness when profiling is urgently needed, provides a reference point for what good performance looks like, and most importantly allows easy and early detection of performance mistakes and regressions, which can be troublesome and expensive in the long run, growing into "deep systemic issues" even if they are harmless when they first appear.

by blt 8y ago

mlthoughts2018 8y ago

My experience writing large scale search engine, NLP and computer vision services, generally as backend web services that have tight performance constraints for other client apps that consume from them, is basically fundamentally the opposite of what you say and what you quoted from the article. It is virtually never the case that profiling reveals a uniformly slow mess of code, and in many cases this doesn’t even make sense as a possibility because you write modular performance tests and profiling tools the same as you write modular unit tests. You would never fire up the profiler and just naively profile a whole backend service all at once, apart from maybe getting extremely coarse latency statistics that only matter in terms of what stakeholder constraints you have to meet. For analyzing cases when you’re not just gathering whole-service stats for someone else, you’d always profile in a more granular, experiment-driven way. > “Developers should have a solid grasp of computer hardware, compilers, interpreters, operating systems, and architecture of high performance software. Obliviousness to these is the source of performance-hostile program designs. If you know these things, you will write better performing code without consciously thinking about it.” While it’s good to know more about more things generally, I think the specific claim that any of this will help you design better code in the first place is totally and emphatically wrong. Instead it leads you down wrong tracks of wasted effort where someone’s “experience and intuition” about necessary optimizations or necessary architectural trade-offs ends up being irrelevant to the eventual end goal, the changing requirements from the sales team, etc. etc. This happens so frequently and clashes so miserably with a basic YAGNI pragmatism that it really is a good heuristic to just say that early focus on optimization characteristics is always premature. In the most optimization-critical applications I’ve worked on, getting primitive and poorly performing prototypes up and running for stakeholders has always been the most critical part to completing the project within the necessary performance constraints. Treating it like an iterative, empirical problem to solve with profiling is absolutely a sacred virtue for pragmatically solving the problem, and carries far, far less risk than precommitting to choices made on the speculative basis of performance characteristics borne out of someone else’s “experience” with performance-intensive concepts or techniques. Such experience really only matters later on, long after you’ve gathered evidence from profiling.

hinkley 8y ago
The performance antagonistic design they’re talking about can’t benefit from what you’re talking about. Worse, it’s usually antagonistic to other things like well-contained modifications. For instance, look at the Big Ball of Mud design. When you are looking at a Big Ball of Mud, these per module performance tests are difficult or impossible to maintain. It’s quite common to see a flame chart with high entropy. Nothing “stands out” after the second or third round of tweaking. Yes there are ways to proceed at this point, but they don’t appear in the literature
gridspy 8y ago
The quote you mention is specifically talking about the architecture / design of the optimal program. Once a design such as a linked-list of polymorphic classes being iterated on in a loop is established, it is hard to refactor. Those with performance experience would understand that this design will have caching, branch prediction and similar issues. However, I totally agree with you that the overall implementation should be driven by iterative coding and profiling. BUT when it is time to choose memory layouts, data models, etc - planning in advance with an eye to performance is very valuable.
dman 8y ago
I think the issue here is that the large scale systems you cite usually have performance being dominated by the relatively high cost of making a network hop. They dominate runtime to such an extent that unless you are writing code with which is >= O(n^2), your runtime is almost entirely dominated by network latency and the number of hops you make. Also many problems in these domains are embarrasingly parallel and you can effectively summon large number of machines to hold state for you in memory. Where it does get interesting is situations like games / garbage collectors where you want to minimise latency. On such problems you usually cant bolt on performance after the fact.
- mlthoughts2018 8y ago
  No, this is not at all what I’m talking about. I’m specifically talking about cases with tight latency requirements after the request has been received. Like receiving a POST request containing many images, and invoking complex image preprocessing with a mix of e.g. opencv and homemade algorithms and then feeding into a deep convolutional neural network. Latencies of 2-3 seconds per request might be the target, and a naive unoptimized implementation of just the backend code (totally unrelated to any network parts) might result in something 10x too slow. In that case, it’s absolutely a best practice to start out with that 10x-too-slow-but-easy-to-implement-for-stakeholders version, and only harden or tighten your homemade computer vision code or use of opencv or some neural network tool after profiling. > “Where it does get interesting is situations like games / garbage collectors where you want to minimise latency. On such problems you usually cant bolt on performance after the fact.” No, that’s exactly the sort of case I’m talking about. It’s not only possibly to refactor for performance after the fact, but dramatically easier to do so, less risky in terms of optimizing for the wrong things, and less prone to re-work.

hinkley 8y ago

We need a name for this. The old advice is to ignore performance becaus it’s not important. When it becomes important there is now very little you can do. You get the low hanging fruit (terrible analogy. Ask a fruit grower how stupid this is), or the tall tent poles. Eventually you have an unending field of nearly identically tall poles, everyone claims victory (we’ve done all we can!). But your sales team is still mad because your product is several times slower than the competing products. What do we call this? Picket fence? Blanket of doom? Death shroud? Forest for the trees? Stupidity?

IncRnd 8y ago
> The old advice is to ignore performance because it’s not important. I think you misunderstood the instructions and anecdotes that you were given about how and when to optimize. Alternatively, your definition of "old" in "old advice" is a very short time in the past.
- hinkley 8y ago
  The only yardstick of teaching that means anything is to look at what people go and do with what you taught them. I spent a big chunk of my early career doing performance analysis work in a language stereotyped as being slow. My thesis was that it was down to developer skill and not the language. By the time someone spoke Knuth's aphorism at me it was people who were standing between me and the strategic goals of the company. Using it as a shield to get out of doing the hard work to match a competitor or customer expectations. As I quickly discovered, there are a whole set of performance optimizations that also improve readability, and that got around the friction with the foot dragging Knuth quoters. There's a host of other things never show up in the literature. It's a shame, really.
  - basilgohar 8y ago
    A lot of this is due to misunderstanding the quote and knowing the bounds where it applies. Premature optimization being the root of all evil means those misapplying the addage are using it as a crutch. Good code is both performant and also easy to scale because of how it's designed (as mentioned elsewhere in this thread). See [0]. It is suspect that most folks ever got it to "Make it right" in the first place. Executive pressure and lack of support from managers needs to be called out as well as contributing to problems with quality of code ad teams oftentimes are not given the chance to make it right under pressure to deliver under unrealistic deadlines. [0] http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast

brightball 8y ago

Honestly, I think you're setting too high of a bar. If developers learn 2 basic things: 1. How SQL queries work 2. Network latency exists. Moving stuff between your database and your server can get expensive. You'll solve _most_ of the issues that I've seen in my career to this point. There are a few big ones that came from abusing API's too, but it's really the same root issue: You're either thinking about how much information is being moved around or you're not.

lomnakkus 8y ago
> You're either thinking about how much information is being moved around or you're not. Not to put too fine a point on it, but I think you meant how often information is being moved around. That's the scenario where latency has a disproportionate effect on performance. (Bandwidth is usually not a problem these days unless you're doing something at the very high end.)
somenewacc 8y ago
For CPU bound stuff, the 80%/20% rule is to learn that an array/vector is typically much faster than a linked list, and this is due to how CPU caches work. (this is thinking about how much information is being moved around, but between the processor and RAM) Also, for parallel stuff: locks tend to be bottlenecks. Writing parallel code is tricky overall. Stick to libraries that provide high level lockless data structures like mpsc queues if possible.
- adrianN 8y ago
  Who uses linked lists? Outside of niche algorithms that splice lots of lists together there are practically zero occasions where linked lists beat dynamic arrays.
  - deleted 8y ago
    [deleted]
  - hvidgaard 8y ago
    Linked lists are very common, especially for non performance critical code. Dynamic arrays either carry a large performance cost at expansion, or is rather complicated to implement in a amortized way. Linked list in comparison is simple, and since it doesn't require a large continues block of memory, the memory pressure is smaller.

didibus 8y ago

I thought it was don't optimize the code prematurely, but at the architechture level, scale and performance must definitly be defined at design time.

MaxBarraclough 8y ago

You've made no mention of what strikes me as a pretty central requirement: at least a basic understanding of data-structures and algorithms. It's a plus to have an understanding of advanced topics like the workings of compilers, but when the foundations are lacking, you're shot. When developers lack a basic grasp of basic data-structures, it harms not only performance, but complexity, readability, maintainability, code-density, and just about everything else we care about.

HelloNurse 8y ago

> Everyone repeats "only optimize after profiling". It's true: if your program is running too slowly, you should always profile. But that rule only applies when your code is already slow. It doesn't actually say anything about the rest of the development process. Except for unusually throwaway or easy to make fast enough projects, software should also be profiled when it runs fast, before there are performance problems. Profiling the application from the beginning ensures readiness when profiling is urgently needed, provides a reference point for what good performance looks like, and most importantly allows easy and early detection of performance mistakes and regressions, which can be troublesome and expensive in the long run, growing into "deep systemic issues" even if they are harmless when they first appear.