5 ms·

Free MIT Course: Performance Engineering of Software Systems

241 points by danielovichdk 3y ago

rashkov 3y ago
Another good course with exercises: https://github.com/dendibakh/perf-ninja https://github.com/dendibakh/perf-ninja
explaininjs 3y ago
I TA’d this course this semester (Fall 2018 - wow the time flies), happy to answer questions. (Though I’m currently backpacking through the Andes so you may need to be patient)
- dmlittle 3y ago
  Thank you for TA'ing the class! I took it in 2015 and TAs really made the class for myself and most of my friends. Is the end of semester Leisserchess competition still going? I believe I heard that the year you TA'd it (might have been a year before or after) a group finally compiled an opening playbook and beat everyone in the class
  - explaininjs 3y ago
    I haven’t followed the course closely, but I’d assume there’s still some variation of the game going on, Charles was quite fond of it. I’m not sure what you mean by “finally complied an opening book”, by the time I was involved with the class opening books were table stakes and we even included code to generate them as part of the starting distribution. They’re somewhat useful, but effective culling of the search space in multithreaded contexts is far more important. The opening book can only ever help for the first few ply, after that the engine which can consistently think a move ahead will likely win. (Indeed the staff designed the rules to optimize for that very property). A quick and informative heuristic function is of course also critical.
- lagniappe 3y ago
  I don't have any questions but do wish you well on your journey through the Andes :) Signing up for the course for fun, thanks!
- flyer_go 3y ago
  Do you have code for the course?
  - explaininjs 3y ago
    Not my place to share it! Though it can be found online by those who seek.
  - stuarto 3y ago
    course repo with code and assignments is at: https://github.com/sourcery-ai-bot/MIT_OpenCourseWare-Performance_Engineering_of_Software_Systems https://github.com/sourcery-ai-bot/MIT_OpenCourseWare-Perfor...
- noitamroftuo 3y ago
  tell me about backpacking through the andes. one question - where do you sleep ?
jstanley 3y ago
The course looks really interesting, but in my experience of "performance engineering" you very rarely need to know this stuff. If you want to improve the performance of some random software system you encounter at work, there is normally much lower-hanging fruit available. First profile the application on as representative a workload as possible. Then fix the accidentally-quadratic loops, stop it from hitting the disk every iteration, add indexes for the worst database queries. If that doesn't at least double your throughput, you are working on an unusually-good application.
- f1shy 3y ago
  This is pretty much stated in the course. Basically is rare that you need it. But when you need it, is good to know.
  - oooyay 3y ago
    More than good to know, sometimes knowing the thresholds of the nuts and bolts is the difference between tuning and rewrite it in a different language or more complex architecture.
- dennis_moore 3y ago
  I think that the term "software system" can have a much broader meaning than the one you are using here.
  - jstanley 3y ago
    For sure. I suppose the unspoken step is "work out which component is the bottleneck". If you're lucky then it's CPU bound and everything runs on the same box and you can just look in htop.
- a1o 3y ago
  > there is normally much lower-hanging fruit Yeap, usually it's more about properly profiling your application on the CPU and memory dimensions, occasionally instrumenting if necessary, experimenting different measurements and tools to have a good picture of your software bottlenecks and where there are gains. Knowing the full picture and where the issues are you can then proceed to these specifics of optimizing your code for performance in these spots, or using a specific library that solves better that problem or what is usually an approach that gives better gain, rethinking knowing the limitations and finding a better design for the problem that part of the code is trying to solve.
_benj 3y ago
[flagged]
- deleted 3y ago
  [deleted]
pclmulqdq 3y ago
As an engineer who has spent a lot of time on performance engineering, I have to say that this course has a lot of the "cool" parts of the job, but not a lot of the actual nuts and bolts of it. Practical performance engineering is primarily about measuring things. In large software systems it's not obvious at all where your hotspots actually are and what you actually should do to take care of them. Further, once you do an optimization, you need to be able to make the right measurements to prove that your optimization actually worked. A lot of optimization also involves parameter tuning (eg how many threads to do what), and that is incredibly dependent on having good measurements. Once you know what to do, the actual optimizations you need to do - even assembly-level tricks - are usually somewhat straightforward in comparison to the measurements you do to get there. If you don't do the measuring, you are going to end up doing a lot of high-effort, low-return work.
- darig 3y ago
  [dead]
- explaininjs 3y ago
  I really don’t know where this mis conception keeps coming from. The course very explicitly makes clear the need for profiling and instrumentation, and the lab write ups require breaking down the impact of each change you made. But that’s all trivial compared to the main content of the class, it can be taught in a single lecture (+ hands on guiding in recitations/homework). After that is where the “ok X specific section is slow, now what?” comes in.
  - pclmulqdq 3y ago
    The mistake in your comment - and probably the course - is the assumption that the measurement, which you limit to only profiling and a vague notion of "instrumentation" (which I assume usually means "reading the performance counters"), is the easy part of performance engineering. Actual performance engineering is rarely as simple as "this huge section of code is slow and hot and hasn't been hyper-optimized yet, let me hyper-optimize it." I have done projects like that in real systems, but they are few and far between compared to projects that involve a lot more measurement and a lot less assembly-level code. Usually, performance engineering either means (1) finding someone doing some O(n^2) shit and diplomatically telling them that they were stupid or (2) finding a relatively subtle and diffuse source of slowness in your system and making equally subtle changes that speed things up significantly. See the following paper for an example of real life performance engineering, where the engineers involved slowed down their own code to create a significant improvement in general application performance: https://www.usenix.org/system/files/osdi21-hunter.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf As another example, performance engineering in trading systems often involves figuring out how to do non-invasive measurement of events that cause systemic tail latency so you can find the bottlenecks and slow parts. If you do the hamfisted things that most engineers think of ("let me chuck performance counter checks everywhere"), you will prevent your system from making money and often destroy the usable signal.
tayo42 3y ago
Looks interesting, wish it touched on system and os level optimizations too.
dwrodri 3y ago
I am currently trying to prep for SWE roles where my focus would be performance / scalability, so this course is extra topical for me. Thank you for sharing. Here are some other resources for people interested in improving their understanding of performance: - "Computer Architecture: A Quantitative Approach" by Hennessy and Patterson. The "Computer Organization" book by the same authors is also good. Both of these resources were extra useful when building deeper intuitions about GPU performance for ML models at work and in graduate school. - CMU's "Deep Learning Systems" Course is hosted online and has YouTube lectures online. While not generally relevant to software performance, it is especially useful for engineers interested in building strong fundamentals that will serve them well when taking ML models into production environments: https://dlsyscourse.org/ https://dlsyscourse.org/ - Compiler Explorer is a tool that allows you easily input some code in and check how the assembly output maps to the source. I think this is exceptionally useful for beginner/intermediate programmers who are familiar with one compiled high-level language and have not been exposed to reading lots of assembly. It is also great for testing how different compiler flags affect assembly output. Many people used to coding in C and C++ probably know about this, but I still run into people who haven't so I share it whenever performance comes up: https://godbolt.org/ https://godbolt.org/
j_not_j 3y ago
See also The Art of Computer Systems Performance Analysis by Jain. The statistical parts are very good: self contained and with good explanations. Perhaps a weakness is there isn't much on distributed systems. He did some Youtube videos as well.

241 points by danielovichdk 3y ago

Another good course with exercises: https://github.com/dendibakh/perf-ninja https://github.com/dendibakh/perf-ninja

I TA’d this course this semester (Fall 2018 - wow the time flies), happy to answer questions. (Though I’m currently backpacking through the Andes so you may need to be patient)

dmlittle 3y ago
Thank you for TA'ing the class! I took it in 2015 and TAs really made the class for myself and most of my friends. Is the end of semester Leisserchess competition still going? I believe I heard that the year you TA'd it (might have been a year before or after) a group finally compiled an opening playbook and beat everyone in the class
- explaininjs 3y ago
  I haven’t followed the course closely, but I’d assume there’s still some variation of the game going on, Charles was quite fond of it. I’m not sure what you mean by “finally complied an opening book”, by the time I was involved with the class opening books were table stakes and we even included code to generate them as part of the starting distribution. They’re somewhat useful, but effective culling of the search space in multithreaded contexts is far more important. The opening book can only ever help for the first few ply, after that the engine which can consistently think a move ahead will likely win. (Indeed the staff designed the rules to optimize for that very property). A quick and informative heuristic function is of course also critical.
lagniappe 3y ago
I don't have any questions but do wish you well on your journey through the Andes :) Signing up for the course for fun, thanks!
flyer_go 3y ago
Do you have code for the course?
- explaininjs 3y ago
  Not my place to share it! Though it can be found online by those who seek.
- stuarto 3y ago
  course repo with code and assignments is at: https://github.com/sourcery-ai-bot/MIT_OpenCourseWare-Performance_Engineering_of_Software_Systems https://github.com/sourcery-ai-bot/MIT_OpenCourseWare-Perfor...
noitamroftuo 3y ago
tell me about backpacking through the andes. one question - where do you sleep ?

jstanley 3y ago

The course looks really interesting, but in my experience of "performance engineering" you very rarely need to know this stuff. If you want to improve the performance of some random software system you encounter at work, there is normally much lower-hanging fruit available. First profile the application on as representative a workload as possible. Then fix the accidentally-quadratic loops, stop it from hitting the disk every iteration, add indexes for the worst database queries. If that doesn't at least double your throughput, you are working on an unusually-good application.

f1shy 3y ago
This is pretty much stated in the course. Basically is rare that you need it. But when you need it, is good to know.
- oooyay 3y ago
  More than good to know, sometimes knowing the thresholds of the nuts and bolts is the difference between tuning and rewrite it in a different language or more complex architecture.
dennis_moore 3y ago
I think that the term "software system" can have a much broader meaning than the one you are using here.
- jstanley 3y ago
  For sure. I suppose the unspoken step is "work out which component is the bottleneck". If you're lucky then it's CPU bound and everything runs on the same box and you can just look in htop.
a1o 3y ago
> there is normally much lower-hanging fruit Yeap, usually it's more about properly profiling your application on the CPU and memory dimensions, occasionally instrumenting if necessary, experimenting different measurements and tools to have a good picture of your software bottlenecks and where there are gains. Knowing the full picture and where the issues are you can then proceed to these specifics of optimizing your code for performance in these spots, or using a specific library that solves better that problem or what is usually an approach that gives better gain, rethinking knowing the limitations and finding a better design for the problem that part of the code is trying to solve.

_benj 3y ago

[flagged]

deleted 3y ago
[deleted]

pclmulqdq 3y ago

As an engineer who has spent a lot of time on performance engineering, I have to say that this course has a lot of the "cool" parts of the job, but not a lot of the actual nuts and bolts of it. Practical performance engineering is primarily about measuring things. In large software systems it's not obvious at all where your hotspots actually are and what you actually should do to take care of them. Further, once you do an optimization, you need to be able to make the right measurements to prove that your optimization actually worked. A lot of optimization also involves parameter tuning (eg how many threads to do what), and that is incredibly dependent on having good measurements. Once you know what to do, the actual optimizations you need to do - even assembly-level tricks - are usually somewhat straightforward in comparison to the measurements you do to get there. If you don't do the measuring, you are going to end up doing a lot of high-effort, low-return work.

darig 3y ago
[dead]
explaininjs 3y ago
I really don’t know where this mis conception keeps coming from. The course very explicitly makes clear the need for profiling and instrumentation, and the lab write ups require breaking down the impact of each change you made. But that’s all trivial compared to the main content of the class, it can be taught in a single lecture (+ hands on guiding in recitations/homework). After that is where the “ok X specific section is slow, now what?” comes in.
- pclmulqdq 3y ago
  The mistake in your comment - and probably the course - is the assumption that the measurement, which you limit to only profiling and a vague notion of "instrumentation" (which I assume usually means "reading the performance counters"), is the easy part of performance engineering. Actual performance engineering is rarely as simple as "this huge section of code is slow and hot and hasn't been hyper-optimized yet, let me hyper-optimize it." I have done projects like that in real systems, but they are few and far between compared to projects that involve a lot more measurement and a lot less assembly-level code. Usually, performance engineering either means (1) finding someone doing some O(n^2) shit and diplomatically telling them that they were stupid or (2) finding a relatively subtle and diffuse source of slowness in your system and making equally subtle changes that speed things up significantly. See the following paper for an example of real life performance engineering, where the engineers involved slowed down their own code to create a significant improvement in general application performance: https://www.usenix.org/system/files/osdi21-hunter.pdf https://www.usenix.org/system/files/osdi21-hunter.pdf As another example, performance engineering in trading systems often involves figuring out how to do non-invasive measurement of events that cause systemic tail latency so you can find the bottlenecks and slow parts. If you do the hamfisted things that most engineers think of ("let me chuck performance counter checks everywhere"), you will prevent your system from making money and often destroy the usable signal.

tayo42 3y ago

Looks interesting, wish it touched on system and os level optimizations too.

dwrodri 3y ago

I am currently trying to prep for SWE roles where my focus would be performance / scalability, so this course is extra topical for me. Thank you for sharing. Here are some other resources for people interested in improving their understanding of performance: - "Computer Architecture: A Quantitative Approach" by Hennessy and Patterson. The "Computer Organization" book by the same authors is also good. Both of these resources were extra useful when building deeper intuitions about GPU performance for ML models at work and in graduate school. - CMU's "Deep Learning Systems" Course is hosted online and has YouTube lectures online. While not generally relevant to software performance, it is especially useful for engineers interested in building strong fundamentals that will serve them well when taking ML models into production environments: https://dlsyscourse.org/ https://dlsyscourse.org/ - Compiler Explorer is a tool that allows you easily input some code in and check how the assembly output maps to the source. I think this is exceptionally useful for beginner/intermediate programmers who are familiar with one compiled high-level language and have not been exposed to reading lots of assembly. It is also great for testing how different compiler flags affect assembly output. Many people used to coding in C and C++ probably know about this, but I still run into people who haven't so I share it whenever performance comes up: https://godbolt.org/ https://godbolt.org/

j_not_j 3y ago

See also The Art of Computer Systems Performance Analysis by Jain. The statistical parts are very good: self contained and with good explanations. Perhaps a weakness is there isn't much on distributed systems. He did some Youtube videos as well.