4 ms·

I’m curious, have you done much profiling? And what tools have you used if you have? This isn’t a judgment against you, it’s a genuine question because I’m wond

by _gabe_ 3y ago

I’m curious, have you done much profiling? And what tools have you used if you have? This isn’t a judgment against you, it’s a genuine question because I’m wondering if perhaps I really am just this inexperienced. But the one thing I’ve learned is I’m always very surprised by where the hotspots in my code come from. I always assume big allocations, or bad cache coherency, or O(1) loops when I could use a hash map will be the killers, but then it turns out it’s something else entirely. Because of this, I’ve just learned to never trust my intuition when it comes to this. Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue. Add to that JIT for certain languages and I don’t even know how you would begin to intuit where a hotspot would be. Edit: And just for additional context, I’ve used Intel’s VTune profiler, Tracy, Optick, and written a few manual benchmarks. Using these tools isn’t always easy, but they give a lot of insights that I wouldn’t pick up on my own.

Mawr 3y ago
It's not usually that hard because the higher level the decision, the more impactful it is on performance and the less a compiler will be able to bail you out. That's why you should start optimizing from the highest level and work your way down until the performance is acceptable. Only once you're at the point of optimizing low-level decisions does profiling really start to make sense. A silly example: it doesn't matter how much you optimize your bubble sort implementation, it'll lose out to the most naively written quicksort, which will lose out to keeping the data always sorted and inserting with binary search, which will lose out to finding out how to make the data inherently sorted, which will lose out to figuring out how to avoid the need for the data to be sorted in the first place. The fastest code is no code at all.
insanitybit 3y ago
Oh yes, quite a bit of profiling, certainly. I've used tools like valgrind and cachegrind, perf, manual instrumentation, JMX, or even just `time`, as well as a dozen others I'm sure. > But the one thing I’ve learned is I’m always very surprised by where the hotspots in my code come from. Are you looking at small or large codebases? The code in the article is quite tiny. The major complexity in terms of understanding that code's performance is really just V8 specific stuff. As someone who only reads casually about V8 I'd be much less comfortable guessing about performance. To be clear, there's plenty of times where I can't guess. Certain languages make it really hard, like if there's a lot of inheritance I can't "glance" at the code at all, I have to dig all around the hierarchies. Extremely annoying, I'll 100% have to benchmark in that sort of language. > Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue. For sure, like I said, the big question here is V8. If this were in a system I'm more familiar with I'd have an easier time telling you which optimizations would or would not work. But even still it's not too hard to guess, sometimes. Knowledge of your runtime is the big thing, I think. For example, reusing an allocation might seem like a trivial optimization but allocations can actually have effects (like failing) so an optimization that removes one would be somewhat aggressive - the situations where that optimization will kick in is going to depend on your compiler/JIT. As an example, it's really easy to find documentation about how escape analysis in the JVM works. There are a few rules and you can sometimes look at code and evaluate those rules in your head. You could use those similar rules for C# or Go or JS but you'd be risking that the rules are different in those languages. Example of an article that goes into some depth on JVM escape analysis and the implications: https://blogs.oracle.com/javamagazine/post/escape-analysis-in-the-hotspot-jit-compiler https://blogs.oracle.com/javamagazine/post/escape-analysis-i... > Because of this, I’ve just learned to never trust my intuition when it comes to this. That's totally fine fwiw. I'm definitely pro "just benchmark it", but I will maintain that: a) Benchmarking and profiling are hard. Getting a representative benchmark of your system under load is particularly difficult and a GC can have wildly different performance characteristics under load. b) It's often not that hard to guess. The more complex the runtime, the more complex the system, the harder it is to guess. > and I don’t even know how you would begin to intuit where a hotspot would be. The short answer is you just learn about how a JIT works just like you learn about how a compiler works. Just to reiterate, I'm very much pro "measure before optimizing". Like, very pro. But also, the article shows that guessing definitely works. And I think guessing is super effective and a fine tool to use at times. Perhaps I've overstated my ability to "just look at code and see the slow bits" - it's obviously very much dependent on the code itself, my familiarity with the underlying components such as the runtime or compiler, and perhaps a bit of luck.
- _gabe_ 3y ago
  Thanks for the clarification and detailed response! I think your final two points are probably the two things I was failing to take into consideration here. I’m used to trying to profile medium-ish systems (like 40-50k lines of code) so that was the context I was bringing into this. In regards to the example listed in the article, you are absolutely right about being able to intuit where the hot spots may come from because it’s such a small isolated piece of code. I’ve still got a long way to go in learning more about profiling, especially when it comes to JITed stuff or things with a heavy runtime. But it’s always nice to hear about how other developers approach profiling (it can feel like black magic to me a lot of the time). Thanks for the article on escape analysis too! I’ll have to check that out in a bit :)
heisenbit 3y ago
The challenge with profiling is that you are getting some results but the interpretation of these results is extremely context dependent and possibly the issue is outside the context you are examining. It may be the tasks, it may be the glue between tasks and the graphs may not always tell the story. I find having some internal assumptions and being able to test them and building deeper mental model of the critical parts helps to flush out some culprits. Being able to at least eliminate some areas as "ok enough" can help to guide the search. It can be an iterative process where finding the right question and tool is more important than later looking at the data.

by _gabe_ 3y ago

Mawr 3y ago

It's not usually that hard because the higher level the decision, the more impactful it is on performance and the less a compiler will be able to bail you out. That's why you should start optimizing from the highest level and work your way down until the performance is acceptable. Only once you're at the point of optimizing low-level decisions does profiling really start to make sense. A silly example: it doesn't matter how much you optimize your bubble sort implementation, it'll lose out to the most naively written quicksort, which will lose out to keeping the data always sorted and inserting with binary search, which will lose out to finding out how to make the data inherently sorted, which will lose out to figuring out how to avoid the need for the data to be sorted in the first place. The fastest code is no code at all.

insanitybit 3y ago

Oh yes, quite a bit of profiling, certainly. I've used tools like valgrind and cachegrind, perf, manual instrumentation, JMX, or even just `time`, as well as a dozen others I'm sure. > But the one thing I’ve learned is I’m always very surprised by where the hotspots in my code come from. Are you looking at small or large codebases? The code in the article is quite tiny. The major complexity in terms of understanding that code's performance is really just V8 specific stuff. As someone who only reads casually about V8 I'd be much less comfortable guessing about performance. To be clear, there's plenty of times where I can't guess. Certain languages make it really hard, like if there's a lot of inheritance I can't "glance" at the code at all, I have to dig all around the hierarchies. Extremely annoying, I'll 100% have to benchmark in that sort of language. > Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue. For sure, like I said, the big question here is V8. If this were in a system I'm more familiar with I'd have an easier time telling you which optimizations would or would not work. But even still it's not too hard to guess, sometimes. Knowledge of your runtime is the big thing, I think. For example, reusing an allocation might seem like a trivial optimization but allocations can actually have effects (like failing) so an optimization that removes one would be somewhat aggressive - the situations where that optimization will kick in is going to depend on your compiler/JIT. As an example, it's really easy to find documentation about how escape analysis in the JVM works. There are a few rules and you can sometimes look at code and evaluate those rules in your head. You could use those similar rules for C# or Go or JS but you'd be risking that the rules are different in those languages. Example of an article that goes into some depth on JVM escape analysis and the implications: https://blogs.oracle.com/javamagazine/post/escape-analysis-in-the-hotspot-jit-compiler https://blogs.oracle.com/javamagazine/post/escape-analysis-i... > Because of this, I’ve just learned to never trust my intuition when it comes to this. That's totally fine fwiw. I'm definitely pro "just benchmark it", but I will maintain that: a) Benchmarking and profiling are hard. Getting a representative benchmark of your system under load is particularly difficult and a GC can have wildly different performance characteristics under load. b) It's often not that hard to guess. The more complex the runtime, the more complex the system, the harder it is to guess. > and I don’t even know how you would begin to intuit where a hotspot would be. The short answer is you just learn about how a JIT works just like you learn about how a compiler works. Just to reiterate, I'm very much pro "measure before optimizing". Like, very pro. But also, the article shows that guessing definitely works. And I think guessing is super effective and a fine tool to use at times. Perhaps I've overstated my ability to "just look at code and see the slow bits" - it's obviously very much dependent on the code itself, my familiarity with the underlying components such as the runtime or compiler, and perhaps a bit of luck.

_gabe_ 3y ago
Thanks for the clarification and detailed response! I think your final two points are probably the two things I was failing to take into consideration here. I’m used to trying to profile medium-ish systems (like 40-50k lines of code) so that was the context I was bringing into this. In regards to the example listed in the article, you are absolutely right about being able to intuit where the hot spots may come from because it’s such a small isolated piece of code. I’ve still got a long way to go in learning more about profiling, especially when it comes to JITed stuff or things with a heavy runtime. But it’s always nice to hear about how other developers approach profiling (it can feel like black magic to me a lot of the time). Thanks for the article on escape analysis too! I’ll have to check that out in a bit :)

heisenbit 3y ago

The challenge with profiling is that you are getting some results but the interpretation of these results is extremely context dependent and possibly the issue is outside the context you are examining. It may be the tasks, it may be the glue between tasks and the graphs may not always tell the story. I find having some internal assumptions and being able to test them and building deeper mental model of the critical parts helps to flush out some culprits. Being able to at least eliminate some areas as "ok enough" can help to guide the search. It can be an iterative process where finding the right question and tool is more important than later looking at the data.