6 ms·

Benchmarking JS

131 points by luu 12y ago

reimertz 12y ago
I found this very interesting since I'm working on something similar where I need to be able to measure how long certain operations take. And I had no clue that js-engines has come this far, optimizing code during runtime this efficiently(if you ignore the bugs.. :)).
- DannyBee 12y ago
  It's actually really well written Some of the optimizations compilers/jits will perform would blow your mind. LICM is a very simple thing to do. There are more advanced things, like http://www.cs.rice.edu/~keith/512/2011/Lectures/L06CLZ-1up.pdf http://www.cs.rice.edu/~keith/512/2011/Lectures/L06CLZ-1up.p... etc That's just on the code movement side too. Most good static compilers can symbolically evaluate even linear systems of equations/recurrences/etc that are solvable, and transform loops that calculate them into constants, etc.
  - mraleph 12y ago
    > LICM is a very simple thing to do. Yeah, it sometimes surprises me how much value the most simple optimizations have and how much they break people's attempts to measure performance. There is another side to this medal which is best expressed in a quote I picked up from a relatively old paper: "A survey of the literature on optimization uncovers many techniques that are hard to understand, harder to implement, more general than necessary and of marginal value. Our approach was to do the most profitable machine independent optimizations in the most profitable way" (A Portable Optimizing Compiler for Modula-2 by Michael L. Powel) In some sense the more microbenchmarks an optimization breaks the bigger its impact on real world code is :)
    - DannyBee 12y ago
      While that paper makes a generally true statement, trying to do the most profitable optimizations in the most profitable way is what dead-ended gcc for many many years :) At some point, you have to step back from what you are doing and actually design and engineer optimizations coherently, not just "do this little thing and that little thing"
  - CountHackulus 12y ago
    LICM is indeed very simple to do, I wrote it for a JIT emulator I was working on a while ago. It turns out that it's not always fast. There's a bunch of weird cases that happen that can destroy your performance. LICM is highly dependent on the quality of your trace recording or method inlining. Not disagreeing with what you wrote at all, just wanted to provide an anecdote.
    - mraleph 12y ago
      Indeed there are some relatively well understood negative consequences to LICM and various other redundancy elimination optimizations: e.g. they increase life-time of values which can have negative impact on register allocation. Another thing is getting LICM right in the presence of conditional control flow within the loop is a non-trivial exercise, in dynamically typed languages this becomes a problem because JIT in general wants to hoist type guards aggressively but it has to account for conditional control flow to avoid weird/unnecessary deopts
      - DannyBee 12y ago
        Yes, LCM is pretty much a disaster in practice, and people have been trying to fix it ever since. The set of things you are worrying about it actually precisely the set of things CLZ solves :)
        mraleph 12y ago
        > The set of things you are worrying about it actually precisely the set of things CLZ solves :) I think the set of things I worry about is slightly different, because they operate in a static environment and I operate in a dynamic one. For example, they don't seem to be worried about the transformation from: loop { if (pred) { // o and o.f are loop invariants Guard(o is X); v = Load(o.f); Use(v) } } to Guard(o is X); v = Load(o.f); loop { if (pred) { Use(v) } } this tranformation might or might not be "beneficial" depending on relation between predicate and the guard - e.g. it can lead to a code which will just deoptimize on the guard before entering the loop. There is no way of knowing this statically, so you just have to take a bet (unless you have clear indications that guard will fail when hoisted) and then undo your decision later.
barsonme 12y ago
Does anybody happen to have the video for the talk? I'd be much obliged. Just the slides leave a bunch of holes I want to fill in. I can only find the videos from 2014.
- mraleph 12y ago
  It was recorded but I don't know when it will be publicly released. This talk is a combination of two separate talks I have given before plus two new examples, based on my recent endeavors. Talks merged here are LXJS 2013 one[1] where I talk about microbenchmarking pitfalls and WebRebels 2014 one[2] where describe how bugs in VMs can affect benchmarking results. If you have any questions about the slides just ask them here or send me an email --- I will try to answer as soon as I can. [1] https://www.youtube.com/watch?v=65-RbBwZQdU https://www.youtube.com/watch?v=65-RbBwZQdU [2] https://webrebels.23video.com/crooked-mirrors-of-performance-by https://webrebels.23video.com/crooked-mirrors-of-performance...
  - tosh 12y ago
    Thanks!
  - barsonme 12y ago
    Okay, thank you! I enjoyed the slides and those talks will give me a lot of study material for tonight. Thanks again!
  - randomfool 12y ago
    Your talks always frighten me when it comes to VM's. I just want predictable performance- is that too much to ask? (Semi-serious question). edit: to clarify. In C++ I write dumb but readable code knowing that the compiler will always perform specific optimizations. In JS it seems that there are many factors resulting in different optimization levels. How do I ensure that the VM does not execute my 'dumb but readable' code literally? Without that guarantee it seems that I have to frequently resort to optimized yet unreadable code.
    - mraleph 12y ago
deleted 12y ago
[deleted]
inglor 12y ago
Can always trust Vyacheslav Egorov to pump out super high quality stuff about v8. His blog http://mrale.ph/ http://mrale.ph/ is full of hidden goodies like Object.create vs function constructor [1] and hidden classes vs jsperf [2]. Lots of good stuff on his SO too. It'd be great if he (hi if you're reading this btw!) could write a bit about TF and the changes being made to v8 internally from an overview point of view. [1] http://mrale.ph/blog/2014/07/30/constructor-vs-objectcreate.html http://mrale.ph/blog/2014/07/30/constructor-vs-objectcreate.... [2] http://mrale.ph/blog/2013/08/14/hidden-classes-vs-jsperf.html http://mrale.ph/blog/2013/08/14/hidden-classes-vs-jsperf.htm...
- inglor 12y ago
  And, since you're reading this (in the other comments) I just want to point out that the presentations you're giving got a lot better (the deck). This one is a lot more witty than the old ones. Props on getting better at presenting - a big step from your old presentations and blog posts, the English is good too here :)
  - gnrlbzik 12y ago
    Agree : ) Deff. awesome experience watching him present. Such talks stick with you through out.
- mraleph 12y ago
  Thanks, I am glad you like my blog :) I might write something about TF but right now it's still not used to compile normal code and the team is just starting on stuff like adaptive optimizations, so it's a bit too early. Also given that I don't work on V8 anymore I always think it's a bit unfair for me to do it --- somebody from V8 team should do it instead, cause my perspective is bound to be slightly different from theirs :)
z3t4 12y ago
You should never "optimize" JavaScript code, like objects vs arrays or methods vs functions. But instead try to optimize your algorithms, like leaving out square root when comparing witch line is longer. I did optimize JavaScript in the past, but it always worked different in different JS implementations and could change in different releases of the same JS engine. So I gave up on it, and it actually makes me more productive not having to think about with method is faster.
- fridek 12y ago
  That is simply not true. To write fast JS you need to understand the basics of the engine, like what a hidden class is, what exactly garbage collector does, how to keep your methods monomorphic or how types work in arrays. I was stunned when recruiting people to see how many just didn't understand why the prototype chains they build are actually creating a separated method instance per each object they allocate. You can see that easily when serious people suggest that an object constructor should return another object with methods on it. Sure, it's valid JS, it has it's advantages and compiler MAY be able to optimize it (last time I checked it didn't), but there will be a limit in what it can do and performance will be impacted. You should perhaps not optimize trivial things like methods vs function, because there is no good underlying reason for them to be much different. You should know when to optimize and what. I love the longer version of Knuth's quote - "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.". The overall architecture, the way you write and organize your code, in my opinion, and in case of JavaScript is likely to be in the 3%. But I also agree not caring too much about it improves productiveness. Most applications really don't need to worry about performance in such depth. Even when they do, it often boils down to keeping one or two core components optimized. The only worry I always have about it, is that when you notice your code style and architecture doesn't play well with JS engines, it's often too late to change it. A friend called me a day before a prod release asking what to do about his program taking a huge chunk of memory for what the profiler described as a "compiled code". It was a large corporate application, with dynamic code loading. We digged into the framework internals just to notice there was no built-in way of deregistering a class once it's created. Since it didn't play nicely with the JS engine, there were many more hidden classes than business models in the system, and most calls were polymorphic. All this compiled code had to be kept somewhere. At this point there was no way he would replace the framework and even quickly fixing it was a challenge.
imaginenore 12y ago
You should optimize your images. These are 6000x4500px 2.4MB PNG images: http://mrale.ph/talks/goto2015/images/intro-slide.png http://mrale.ph/talks/goto2015/images/intro-slide.png http://mrale.ph/talks/goto2015/images/last-slide.png http://mrale.ph/talks/goto2015/images/last-slide.png
- mraleph 12y ago
  Oops, sorry for that. Forgot to shrink them before publishing. Will fix as soon as I get to a place with a stable internet connection - traveling right now. upd. done, scaled down to 1024x768
gnrlbzik 12y ago
This was a great talk and funny one at that : ) Great perspective on what to really be concerned with. There were other great talks at #GOTOChgo but this one is one of the best ones : )
xenofiend 12y ago
How is the acronym LICM pronounced?
- mraleph 12y ago
  I always pronounce it as /līkm/, but I am not a native speaker so I can't be sure about correct pronunciation.

131 points by luu 12y ago

I found this very interesting since I'm working on something similar where I need to be able to measure how long certain operations take. And I had no clue that js-engines has come this far, optimizing code during runtime this efficiently(if you ignore the bugs.. :)).

DannyBee 12y ago
It's actually really well written Some of the optimizations compilers/jits will perform would blow your mind. LICM is a very simple thing to do. There are more advanced things, like http://www.cs.rice.edu/~keith/512/2011/Lectures/L06CLZ-1up.pdf http://www.cs.rice.edu/~keith/512/2011/Lectures/L06CLZ-1up.p... etc That's just on the code movement side too. Most good static compilers can symbolically evaluate even linear systems of equations/recurrences/etc that are solvable, and transform loops that calculate them into constants, etc.
- mraleph 12y ago
  > LICM is a very simple thing to do. Yeah, it sometimes surprises me how much value the most simple optimizations have and how much they break people's attempts to measure performance. There is another side to this medal which is best expressed in a quote I picked up from a relatively old paper: "A survey of the literature on optimization uncovers many techniques that are hard to understand, harder to implement, more general than necessary and of marginal value. Our approach was to do the most profitable machine independent optimizations in the most profitable way" (A Portable Optimizing Compiler for Modula-2 by Michael L. Powel) In some sense the more microbenchmarks an optimization breaks the bigger its impact on real world code is :)
  - DannyBee 12y ago
    While that paper makes a generally true statement, trying to do the most profitable optimizations in the most profitable way is what dead-ended gcc for many many years :) At some point, you have to step back from what you are doing and actually design and engineer optimizations coherently, not just "do this little thing and that little thing"
- CountHackulus 12y ago
  LICM is indeed very simple to do, I wrote it for a JIT emulator I was working on a while ago. It turns out that it's not always fast. There's a bunch of weird cases that happen that can destroy your performance. LICM is highly dependent on the quality of your trace recording or method inlining. Not disagreeing with what you wrote at all, just wanted to provide an anecdote.
  - mraleph 12y ago
    Indeed there are some relatively well understood negative consequences to LICM and various other redundancy elimination optimizations: e.g. they increase life-time of values which can have negative impact on register allocation. Another thing is getting LICM right in the presence of conditional control flow within the loop is a non-trivial exercise, in dynamically typed languages this becomes a problem because JIT in general wants to hoist type guards aggressively but it has to account for conditional control flow to avoid weird/unnecessary deopts
    - DannyBee 12y ago
      Yes, LCM is pretty much a disaster in practice, and people have been trying to fix it ever since. The set of things you are worrying about it actually precisely the set of things CLZ solves :)
      - mraleph 12y ago
        > The set of things you are worrying about it actually precisely the set of things CLZ solves :) I think the set of things I worry about is slightly different, because they operate in a static environment and I operate in a dynamic one. For example, they don't seem to be worried about the transformation from: loop { if (pred) { // o and o.f are loop invariants Guard(o is X); v = Load(o.f); Use(v) } } to Guard(o is X); v = Load(o.f); loop { if (pred) { Use(v) } } this tranformation might or might not be "beneficial" depending on relation between predicate and the guard - e.g. it can lead to a code which will just deoptimize on the guard before entering the loop. There is no way of knowing this statically, so you just have to take a bet (unless you have clear indications that guard will fail when hoisted) and then undo your decision later.

barsonme 12y ago

Does anybody happen to have the video for the talk? I'd be much obliged. Just the slides leave a bunch of holes I want to fill in. I can only find the videos from 2014.

mraleph 12y ago
It was recorded but I don't know when it will be publicly released. This talk is a combination of two separate talks I have given before plus two new examples, based on my recent endeavors. Talks merged here are LXJS 2013 one[1] where I talk about microbenchmarking pitfalls and WebRebels 2014 one[2] where describe how bugs in VMs can affect benchmarking results. If you have any questions about the slides just ask them here or send me an email --- I will try to answer as soon as I can. [1] https://www.youtube.com/watch?v=65-RbBwZQdU https://www.youtube.com/watch?v=65-RbBwZQdU [2] https://webrebels.23video.com/crooked-mirrors-of-performance-by https://webrebels.23video.com/crooked-mirrors-of-performance...
- tosh 12y ago
  Thanks!
- barsonme 12y ago
  Okay, thank you! I enjoyed the slides and those talks will give me a lot of study material for tonight. Thanks again!
- randomfool 12y ago
  Your talks always frighten me when it comes to VM's. I just want predictable performance- is that too much to ask? (Semi-serious question). edit: to clarify. In C++ I write dumb but readable code knowing that the compiler will always perform specific optimizations. In JS it seems that there are many factors resulting in different optimization levels. How do I ensure that the VM does not execute my 'dumb but readable' code literally? Without that guarantee it seems that I have to frequently resort to optimized yet unreadable code.
  - mraleph 12y ago

deleted 12y ago

[deleted]

inglor 12y ago

Can always trust Vyacheslav Egorov to pump out super high quality stuff about v8. His blog http://mrale.ph/ http://mrale.ph/ is full of hidden goodies like Object.create vs function constructor [1] and hidden classes vs jsperf [2]. Lots of good stuff on his SO too. It'd be great if he (hi if you're reading this btw!) could write a bit about TF and the changes being made to v8 internally from an overview point of view. [1] http://mrale.ph/blog/2014/07/30/constructor-vs-objectcreate.html http://mrale.ph/blog/2014/07/30/constructor-vs-objectcreate.... [2] http://mrale.ph/blog/2013/08/14/hidden-classes-vs-jsperf.html http://mrale.ph/blog/2013/08/14/hidden-classes-vs-jsperf.htm...

inglor 12y ago
And, since you're reading this (in the other comments) I just want to point out that the presentations you're giving got a lot better (the deck). This one is a lot more witty than the old ones. Props on getting better at presenting - a big step from your old presentations and blog posts, the English is good too here :)
- gnrlbzik 12y ago
  Agree : ) Deff. awesome experience watching him present. Such talks stick with you through out.
mraleph 12y ago
Thanks, I am glad you like my blog :) I might write something about TF but right now it's still not used to compile normal code and the team is just starting on stuff like adaptive optimizations, so it's a bit too early. Also given that I don't work on V8 anymore I always think it's a bit unfair for me to do it --- somebody from V8 team should do it instead, cause my perspective is bound to be slightly different from theirs :)

z3t4 12y ago

You should never "optimize" JavaScript code, like objects vs arrays or methods vs functions. But instead try to optimize your algorithms, like leaving out square root when comparing witch line is longer. I did optimize JavaScript in the past, but it always worked different in different JS implementations and could change in different releases of the same JS engine. So I gave up on it, and it actually makes me more productive not having to think about with method is faster.

fridek 12y ago
That is simply not true. To write fast JS you need to understand the basics of the engine, like what a hidden class is, what exactly garbage collector does, how to keep your methods monomorphic or how types work in arrays. I was stunned when recruiting people to see how many just didn't understand why the prototype chains they build are actually creating a separated method instance per each object they allocate. You can see that easily when serious people suggest that an object constructor should return another object with methods on it. Sure, it's valid JS, it has it's advantages and compiler MAY be able to optimize it (last time I checked it didn't), but there will be a limit in what it can do and performance will be impacted. You should perhaps not optimize trivial things like methods vs function, because there is no good underlying reason for them to be much different. You should know when to optimize and what. I love the longer version of Knuth's quote - "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.". The overall architecture, the way you write and organize your code, in my opinion, and in case of JavaScript is likely to be in the 3%. But I also agree not caring too much about it improves productiveness. Most applications really don't need to worry about performance in such depth. Even when they do, it often boils down to keeping one or two core components optimized. The only worry I always have about it, is that when you notice your code style and architecture doesn't play well with JS engines, it's often too late to change it. A friend called me a day before a prod release asking what to do about his program taking a huge chunk of memory for what the profiler described as a "compiled code". It was a large corporate application, with dynamic code loading. We digged into the framework internals just to notice there was no built-in way of deregistering a class once it's created. Since it didn't play nicely with the JS engine, there were many more hidden classes than business models in the system, and most calls were polymorphic. All this compiled code had to be kept somewhere. At this point there was no way he would replace the framework and even quickly fixing it was a challenge.

imaginenore 12y ago

You should optimize your images. These are 6000x4500px 2.4MB PNG images: http://mrale.ph/talks/goto2015/images/intro-slide.png http://mrale.ph/talks/goto2015/images/intro-slide.png http://mrale.ph/talks/goto2015/images/last-slide.png http://mrale.ph/talks/goto2015/images/last-slide.png

mraleph 12y ago
Oops, sorry for that. Forgot to shrink them before publishing. Will fix as soon as I get to a place with a stable internet connection - traveling right now. upd. done, scaled down to 1024x768

gnrlbzik 12y ago

This was a great talk and funny one at that : ) Great perspective on what to really be concerned with. There were other great talks at #GOTOChgo but this one is one of the best ones : )

xenofiend 12y ago

How is the acronym LICM pronounced?

mraleph 12y ago
I always pronounce it as /līkm/, but I am not a native speaker so I can't be sure about correct pronunciation.