4 ms·

Crazy that she basically implemented a fully functioning JIT compiler that performs on par with the established commercial Javascript interpreters out there on

by tinco 12y ago

Crazy that she basically implemented a fully functioning JIT compiler that performs on par with the established commercial Javascript interpreters out there on many fronts in only two years using novel techniques, but reviewers reject her papers because they are unconvinced the novel technique has advantages over existing techniques? How does that even matter? If someone discovers a new way of implementing compilers efficiently, then just that has scientific merit. Whether the new technique has different performance characteristics surely is simply ground for further research? Anyway, awesome that at least the engineering community is very interested in this work of hers. edit: I noticed the compiler is written in D, awesome choice of tech!

mkaufmann 12y ago
Concerning your critique of the in your opinion overly hard scientific review process. According to her blog the paper was rejected because: "Reviewers at conferences [...] have been very skeptical, and pressed us multiple times to produce some kind of comparison of basic block versioning against tracing."[1] I think this is a very valid concern. When you describe a new scientific approach (Basic Block Versioning) you should compare to the state of the art. Otherwise it is very hard for the reader to judge the merit of the new approach. However I agree with your sentiment that a new approach should not only be judged by performance numbers (there is actually a nice article going deeper on this topic on databasearchitects[2]). But there should at least be a more thorough theoretical discussion than only 3-4 sentences. Benchmarks comparing to existing approaches in this case can help to show pathological cases which might indicate weaknesses of the approach or to empirically demonstrate the feasibility of the new approach. [1] http://pointersgonewild.wordpress.com/2014/11/14/the-fastest-for-in-loops-in-the-whole-wide-web/ http://pointersgonewild.wordpress.com/2014/11/14/the-fastest... [2] http://databasearchitects.blogspot.de/2014/09/experiments-hurt-review-process.html http://databasearchitects.blogspot.de/2014/09/experiments-hu... EDIT: Many conferences also allow publishing papers without a deep comparison to exisiting research in the industrial session. This allows demonstrations of interesting implementation variants or system choices EDIT 2: The review critisim in older blog posts "Conference reviewers criticized us for not discussing compilation times, and raised the issue that perhaps basic block versioning could drastically increase compilation times." is also very valid for a jit compiler. Again discussion does not have to mean that you have to be faster than all existing systems. Paper acceptance is always a little bit a random process, but at least in this case the review comments are valid from my point of view and her phd advisor should probably have detected these problems in proofreading before submitting. I really hope that the paper will finally be accepted!
srean 12y ago
Unfortunate as it is, peer review is quite broken and it is not quite clear how to fix it. Even if the intended goal is otherwise, in practice, peer review in CS turns into an adversarial exercise, with some differences in levels. Rather than inspecting a submission and thinking if there is anything in the paper that might be of interest to the community, could the paper spur real progress, the review process usually boils down to: can I quickly skim to find something / anything with which I can shoot down this paper and get away with it and get done with this review. This of course is a generalization, not implying that this is what happened here. Unfortunately, the prototypical reviewer has become more like a prosecutor. There are various reasons why this has become the norm, people are aware of it, some are even trying hard to figure out ways to make it better, whereas some believe this is exactly how it should be. If accuracy of prediction was a concern, Copernican model would never have been accepted. It took years of polish to make the then new model attain the accuracy of the then prevalent model. Some would argue that such models (the Copernican for example) should remain unpublished and under covers till they beat state of the art. I personally do not agree with this view. I would rather have a reasonably well baked but not yet perfect model out in the ether soon so that the model can enlist / recruit other people to work on it. It really boils down to where in the spectrum should the community operate: (i) spammed by advances of questionable merit, with the community spending effort to winnow the good from the bad or (ii) making sure that only the rock solid ones get through, at the risk of delaying or extinguishing useful advance and waiting for a rediscovery.
- seanmcdirmid 12y ago
  I'm guessing the paper was submitted to PLDI or POPL (wordpress is blocked here, so I can't read the context), the tier one PL conferences. The communities are not adversarial so much as they are enamored with science. I think from that perspective, rejection is understandable: they want more numbers and comparisons, and might accept the paper even if these numbers are not very enlightening (call it going through the motions). You also don't get many points just for wild experimentation. For the same reason, many of the papers accepted seem very incremental and disappointing from a "new idea" perspective; and once you get the paper in, you have to go to the conference (I'm not a PLDI/POPL person)! One can always publish in a "new idea" PL conference (like my personal choice, Onward!, and there is even a new one called SNAPL from the people that brought us PLDI and POPL), but the points you get for publishing at those venues aren't as much as the true technical conferences (I don't care, since I have a job, but for a grad student, its a big deal). My advice would be to suck it up and write a good PLDI paper. Be very thorough and honest with numbers and comparisons, and try to do it "right" (no pointless numbers that don't contribute to the story even if they might "work"). Then with a little bit of luck in PC reviewer selection (most PLDI reviewers are reasonable, some are not), the paper would probably get in. And if they do it right, it could be a really good paper that are lacking these days. Actually, that is another point: sometimes rejection is good, because the paper that gets in later is much much better. Patience.
hnov 12y ago
Could it be that this technique is not as broadly applicable as the existing JIT techniques? Would it be possible to use the approach taken with Higgs to implement a language with a more open ended object system such as ruby? In JavaScript there's an upper bound on what a + b could mean, does BBV work well for more general method dispatch?
- tachyonbeam 12y ago
  No reason it wouldn't work for ruby. It's about propagating type information. The way specific operators are implemented makes not much difference.
gsnedders 12y ago
Note she's only comparing it with TraceMonkey (which is now gone from SpiderMonkey) and the initial V8 compiler (i.e., not the optimising Crankshaft compiler). This makes competing much easier. It's about the timespan I'd have expected — Carakan (the second latest in-browser JS VM; Chakra is the latest, but relatively little is known about it) was essentially feature complete and competitive on perf after nine months of development (with 3.5 developers, and an increasing number of QA from 0.5 up to 2). Carakan's big thing was finding and fixing ever more obscure bugs (especially in the compiler), taking almost the same time again.

by tinco 12y ago

mkaufmann 12y ago

Concerning your critique of the in your opinion overly hard scientific review process. According to her blog the paper was rejected because: "Reviewers at conferences [...] have been very skeptical, and pressed us multiple times to produce some kind of comparison of basic block versioning against tracing."[1] I think this is a very valid concern. When you describe a new scientific approach (Basic Block Versioning) you should compare to the state of the art. Otherwise it is very hard for the reader to judge the merit of the new approach. However I agree with your sentiment that a new approach should not only be judged by performance numbers (there is actually a nice article going deeper on this topic on databasearchitects[2]). But there should at least be a more thorough theoretical discussion than only 3-4 sentences. Benchmarks comparing to existing approaches in this case can help to show pathological cases which might indicate weaknesses of the approach or to empirically demonstrate the feasibility of the new approach. [1] http://pointersgonewild.wordpress.com/2014/11/14/the-fastest-for-in-loops-in-the-whole-wide-web/ http://pointersgonewild.wordpress.com/2014/11/14/the-fastest... [2] http://databasearchitects.blogspot.de/2014/09/experiments-hurt-review-process.html http://databasearchitects.blogspot.de/2014/09/experiments-hu... EDIT: Many conferences also allow publishing papers without a deep comparison to exisiting research in the industrial session. This allows demonstrations of interesting implementation variants or system choices EDIT 2: The review critisim in older blog posts "Conference reviewers criticized us for not discussing compilation times, and raised the issue that perhaps basic block versioning could drastically increase compilation times." is also very valid for a jit compiler. Again discussion does not have to mean that you have to be faster than all existing systems. Paper acceptance is always a little bit a random process, but at least in this case the review comments are valid from my point of view and her phd advisor should probably have detected these problems in proofreading before submitting. I really hope that the paper will finally be accepted!

srean 12y ago

Unfortunate as it is, peer review is quite broken and it is not quite clear how to fix it. Even if the intended goal is otherwise, in practice, peer review in CS turns into an adversarial exercise, with some differences in levels. Rather than inspecting a submission and thinking if there is anything in the paper that might be of interest to the community, could the paper spur real progress, the review process usually boils down to: can I quickly skim to find something / anything with which I can shoot down this paper and get away with it and get done with this review. This of course is a generalization, not implying that this is what happened here. Unfortunately, the prototypical reviewer has become more like a prosecutor. There are various reasons why this has become the norm, people are aware of it, some are even trying hard to figure out ways to make it better, whereas some believe this is exactly how it should be. If accuracy of prediction was a concern, Copernican model would never have been accepted. It took years of polish to make the then new model attain the accuracy of the then prevalent model. Some would argue that such models (the Copernican for example) should remain unpublished and under covers till they beat state of the art. I personally do not agree with this view. I would rather have a reasonably well baked but not yet perfect model out in the ether soon so that the model can enlist / recruit other people to work on it. It really boils down to where in the spectrum should the community operate: (i) spammed by advances of questionable merit, with the community spending effort to winnow the good from the bad or (ii) making sure that only the rock solid ones get through, at the risk of delaying or extinguishing useful advance and waiting for a rediscovery.

seanmcdirmid 12y ago
I'm guessing the paper was submitted to PLDI or POPL (wordpress is blocked here, so I can't read the context), the tier one PL conferences. The communities are not adversarial so much as they are enamored with science. I think from that perspective, rejection is understandable: they want more numbers and comparisons, and might accept the paper even if these numbers are not very enlightening (call it going through the motions). You also don't get many points just for wild experimentation. For the same reason, many of the papers accepted seem very incremental and disappointing from a "new idea" perspective; and once you get the paper in, you have to go to the conference (I'm not a PLDI/POPL person)! One can always publish in a "new idea" PL conference (like my personal choice, Onward!, and there is even a new one called SNAPL from the people that brought us PLDI and POPL), but the points you get for publishing at those venues aren't as much as the true technical conferences (I don't care, since I have a job, but for a grad student, its a big deal). My advice would be to suck it up and write a good PLDI paper. Be very thorough and honest with numbers and comparisons, and try to do it "right" (no pointless numbers that don't contribute to the story even if they might "work"). Then with a little bit of luck in PC reviewer selection (most PLDI reviewers are reasonable, some are not), the paper would probably get in. And if they do it right, it could be a really good paper that are lacking these days. Actually, that is another point: sometimes rejection is good, because the paper that gets in later is much much better. Patience.

hnov 12y ago

Could it be that this technique is not as broadly applicable as the existing JIT techniques? Would it be possible to use the approach taken with Higgs to implement a language with a more open ended object system such as ruby? In JavaScript there's an upper bound on what a + b could mean, does BBV work well for more general method dispatch?

tachyonbeam 12y ago
No reason it wouldn't work for ruby. It's about propagating type information. The way specific operators are implemented makes not much difference.

gsnedders 12y ago

Note she's only comparing it with TraceMonkey (which is now gone from SpiderMonkey) and the initial V8 compiler (i.e., not the optimising Crankshaft compiler). This makes competing much easier. It's about the timespan I'd have expected — Carakan (the second latest in-browser JS VM; Chakra is the latest, but relatively little is known about it) was essentially feature complete and competitive on perf after nine months of development (with 3.5 developers, and an increasing number of QA from 0.5 up to 2). Carakan's big thing was finding and fixing ever more obscure bugs (especially in the compiler), taking almost the same time again.