4 ms·

For this research to be useful, the authors should switch to large benchmarks. VMs win in the average. Therefore, VMs have great cases, average cases, and crap

For this research to be useful, the authors should switch to large benchmarks. VMs win in the average. Therefore, VMs have great cases, average cases, and crappy cases. The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average (or conversely praising him for his biggest wins when they’re behind in the average). Source: I build VMs for a living.

stcredzero 4y ago
The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. I think this is very much the lens of the VM developer. From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average The losses are still losses. Maybe we should try to eliminate the "gambling?"
- pizlonator 4y ago
  VMs are fundamentally about gambling. If you knew statically what to do in the compiler then you’d run an ahead of time (AOT) compiler. VMs are for those cases where an AOT would have been slower, and empirically, languages like JS, Java, and Lua perform much better with a dynamic VM than with an AOT and that’s why we use VMs. So, eliminating gambling is like saying we should just do AOTs. We could but then we’d be slower. > From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling.
  - stcredzero 4y ago
    A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling. I should think "not exactly." A professional gambler would take a look at each game, and determine if the expected payoff makes it worth playing. Here is where the analogy falls down. Not every analogue of "game" is the same! I worked for awhile for a company that was effectively a JIT VM vendor, and we found scenarios where our JIT could outperform things like a naively written YACC/LEX compiler. (Or a web templating engine. If we cranked up the size of the JIT code cache that ran at speeds expected of a C implementation.)
    - pizlonator 4y ago
      No need to overthink the analogy. Pro gamblers choose their games while VM developers don’t. The analogy works for those games the gambler did happen to play. Let me give you more concrete data. On JSC we did tune for microbenchmarks in addition to macrobenchmarks. But the microbenchmarks would show pathologies not seen in the macrobenchmarks, those pathologies were unpredictable (you breathe on the VM and the behavior changes), and anyway real programs were more like macrobenchmarks than microbenchmarks. So - based on experience we switched to using the microbenchmarks only if they reliably reproduced behavior that we first observed in a macrobenchmarks. Got a macrobenchmark regression? See if any microbenchmarks also regressed in hopes of finding something easier to study. Otherwise the microbenchmarks were just whack-a-mole - if you focus on one of them you end up making changes that hurt real stuff. Sort of like if a gambler focused too much on one game, they’d end up losing in the average.
      - stcredzero 4y ago
        Pro gamblers choose their games while VM developers don’t. Again, your answer is very informed, but very much in the VM developer POV. Pro gamblers can and should choose their games. Pro developers can and should also choose their games to make sure their expected payoff works out. While VM developers can "play" from the POV of the "house," and handle things in aggregate, we applicaiton developers choosing to use a VM or not are very much invested in our "next game."
dan-robertson 4y ago
Bigger benchmarks would probably be better but don’t these papers show that the thing programmers commonly believe (that vms will warm up and then run faster, especially for microbenchmarks) is pretty untrue? VMs don’t always or even very often reach a steady state and that steady state may not be faster, even for microbenchmarks. I’m surprised that the author of the post (who I generally find writes interesting, thoughtful and seemingly correct things) feels that the research talked about in this article was pushing the state of the art in reliable benchmarking and understanding vm warmup, but that your comment seems mostly dismissive. I would especially expect a VM developer to be interested in more reliable benchmarking. Do you think that that assessment by the author was incorrect or that these things are widely known among vim developers? I also don’t really understand your comment, eg you claim that VMs win on average but the numbers in the article show that, even for a case which ought to be good for a JIT (I might be wrong here?) with a small very hot section, VMs on average fail to end up in a steady state with better performance than the interpreted case. Is the idea that the good steady state performance you get some of the time will average out with the poor or inconsistent performance you get other times to get good performance? I can sort of see that argument being made about a big system with many instances but I don’t think it’s a correct one because in practice many systems like that will block on the slowest part and so have worse tail performance than a single component. I guess another idea could be that a real program has many parts and if 10% reach a steady fast state while the rest are inconsistent or slower, then on average the program can still be faster. But I don’t really buy this because I expect there are a few places that will matter far more and so you don’t average out very well, but my prior could be badly wrong here.
- pizlonator 4y ago
  The authors cherry-picked benchmarks where the VM goes off the rails. Because warmup is probabilistic, there must be cases where it goes off the rails. So yeah, I’m dismissive.
  - ltratt 4y ago
    Hello Filip -- I hope life is treating you well! I'm happy to clarify a couple of things that might be useful. First, VM authors I've discussed this with over the years seem roughly split down the middle on microbenchmarks. Some very much agree with your perspective that small benchmarks are misleading. Some, though, were very surprised at the quantity and nature of what we found. Indeed, I discovered a small number had not only noticed similar problems in the past but spent huge amounts of time trying to fix them. There are many people who I, and I suspect you, admire, in both camps: this seems like something upon which reasonable people can differ. Perhaps future research will provide more clarity in this regard. Second, for BBKMT we used the first benchmarks we tried, so there was absolutely no cherry picking going on. Indeed, we arguably biased the whole experiment in favour of VMs (our paper details why and how we did so). Since TCPT uses 600 (well, 586...) benchmarks it seems unlikely to me that they cherry picked either. "Cherry picking" is, to my mind, a serious accusation, since it would suggest we did not do our research in good faith. I hope I can put your mind at rest on that matter.
fulafel 4y ago
This seems common but not a necessary property of course. If predictable performance was a design constraint, you could do it. And there's a spectrum between "maximally predictable" and "very unpredictable". Unpredictability could eg result in a need to more generous overprovisioning infrastructure and have less deterministic scaling, or some users having a game run slowly despite meeting hw requirements, so sometimes it might be good to have deterministic but bit slower on average execution.

stcredzero 4y ago

The crappy cases will always exist, and that’s known to VM architects and it’s not a bug. I think this is very much the lens of the VM developer. From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? Essentially this work is like criticizing a professional gambler for his biggest losses when the gambler is ahead in the average The losses are still losses. Maybe we should try to eliminate the "gambling?"

pizlonator 4y ago
VMs are fundamentally about gambling. If you knew statically what to do in the compiler then you’d run an ahead of time (AOT) compiler. VMs are for those cases where an AOT would have been slower, and empirically, languages like JS, Java, and Lua perform much better with a dynamic VM than with an AOT and that’s why we use VMs. So, eliminating gambling is like saying we should just do AOTs. We could but then we’d be slower. > From a different POV, why isn't it valid to ask, "Why can't this also be fast?" Why is that a feature and not a bug? A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling.
- stcredzero 4y ago
  A professional gambler ought not ask this question. A professional gambler ought instead ask: how am I doing in the average? Hence the flaw in this research. It’s bad gambling. I should think "not exactly." A professional gambler would take a look at each game, and determine if the expected payoff makes it worth playing. Here is where the analogy falls down. Not every analogue of "game" is the same! I worked for awhile for a company that was effectively a JIT VM vendor, and we found scenarios where our JIT could outperform things like a naively written YACC/LEX compiler. (Or a web templating engine. If we cranked up the size of the JIT code cache that ran at speeds expected of a C implementation.)
  - pizlonator 4y ago
    No need to overthink the analogy. Pro gamblers choose their games while VM developers don’t. The analogy works for those games the gambler did happen to play. Let me give you more concrete data. On JSC we did tune for microbenchmarks in addition to macrobenchmarks. But the microbenchmarks would show pathologies not seen in the macrobenchmarks, those pathologies were unpredictable (you breathe on the VM and the behavior changes), and anyway real programs were more like macrobenchmarks than microbenchmarks. So - based on experience we switched to using the microbenchmarks only if they reliably reproduced behavior that we first observed in a macrobenchmarks. Got a macrobenchmark regression? See if any microbenchmarks also regressed in hopes of finding something easier to study. Otherwise the microbenchmarks were just whack-a-mole - if you focus on one of them you end up making changes that hurt real stuff. Sort of like if a gambler focused too much on one game, they’d end up losing in the average.
    - stcredzero 4y ago
      Pro gamblers choose their games while VM developers don’t. Again, your answer is very informed, but very much in the VM developer POV. Pro gamblers can and should choose their games. Pro developers can and should also choose their games to make sure their expected payoff works out. While VM developers can "play" from the POV of the "house," and handle things in aggregate, we applicaiton developers choosing to use a VM or not are very much invested in our "next game."

dan-robertson 4y ago

Bigger benchmarks would probably be better but don’t these papers show that the thing programmers commonly believe (that vms will warm up and then run faster, especially for microbenchmarks) is pretty untrue? VMs don’t always or even very often reach a steady state and that steady state may not be faster, even for microbenchmarks. I’m surprised that the author of the post (who I generally find writes interesting, thoughtful and seemingly correct things) feels that the research talked about in this article was pushing the state of the art in reliable benchmarking and understanding vm warmup, but that your comment seems mostly dismissive. I would especially expect a VM developer to be interested in more reliable benchmarking. Do you think that that assessment by the author was incorrect or that these things are widely known among vim developers? I also don’t really understand your comment, eg you claim that VMs win on average but the numbers in the article show that, even for a case which ought to be good for a JIT (I might be wrong here?) with a small very hot section, VMs on average fail to end up in a steady state with better performance than the interpreted case. Is the idea that the good steady state performance you get some of the time will average out with the poor or inconsistent performance you get other times to get good performance? I can sort of see that argument being made about a big system with many instances but I don’t think it’s a correct one because in practice many systems like that will block on the slowest part and so have worse tail performance than a single component. I guess another idea could be that a real program has many parts and if 10% reach a steady fast state while the rest are inconsistent or slower, then on average the program can still be faster. But I don’t really buy this because I expect there are a few places that will matter far more and so you don’t average out very well, but my prior could be badly wrong here.

pizlonator 4y ago
The authors cherry-picked benchmarks where the VM goes off the rails. Because warmup is probabilistic, there must be cases where it goes off the rails. So yeah, I’m dismissive.
- ltratt 4y ago
  Hello Filip -- I hope life is treating you well! I'm happy to clarify a couple of things that might be useful. First, VM authors I've discussed this with over the years seem roughly split down the middle on microbenchmarks. Some very much agree with your perspective that small benchmarks are misleading. Some, though, were very surprised at the quantity and nature of what we found. Indeed, I discovered a small number had not only noticed similar problems in the past but spent huge amounts of time trying to fix them. There are many people who I, and I suspect you, admire, in both camps: this seems like something upon which reasonable people can differ. Perhaps future research will provide more clarity in this regard. Second, for BBKMT we used the first benchmarks we tried, so there was absolutely no cherry picking going on. Indeed, we arguably biased the whole experiment in favour of VMs (our paper details why and how we did so). Since TCPT uses 600 (well, 586...) benchmarks it seems unlikely to me that they cherry picked either. "Cherry picking" is, to my mind, a serious accusation, since it would suggest we did not do our research in good faith. I hope I can put your mind at rest on that matter.

fulafel 4y ago

This seems common but not a necessary property of course. If predictable performance was a design constraint, you could do it. And there's a spectrum between "maximally predictable" and "very unpredictable". Unpredictability could eg result in a need to more generous overprovisioning infrastructure and have less deterministic scaling, or some users having a game run slowly despite meeting hw requirements, so sometimes it might be good to have deterministic but bit slower on average execution.