3 ms·

I had also the feeling the "benchmark" is a joke. Even if it takes some seconds to print hello, speed is only important when you run big programs with million o

I had also the feeling the "benchmark" is a joke. Even if it takes some seconds to print hello, speed is only important when you run big programs with million of operations.

pizlonator 10y ago
If it takes a system 300ms to compile a function that prints "hello" then imagine how long it would take to compile a function that does something more sophisticated. The "hello" benchmark is a fantastic benchmark for production-strength JIT-based runtimes because it tells you how long it takes for your system to warm up enough to be able to print something to the screen. You don't want this to be long. I prefer for "hello" to take about 10ms or less. In WebKit we have a few benchmarks that are the moral equivalent of "hello" and these are some of our most important benchmarks. The reason why warm-up time is so insanely important is that any programming language, no matter what the domain is, will be used for the following things eventually: - Tests. - Data definitions in your language (moral equivalent of JSONP - every language/runtime has to deal with something like this). - Initialization. All of these things comprise run-once code that tends to be large. All of these things are important, even in math languages. The glorious "hello" benchmark is an excellent proxy for the performance of these kinds of code. So, I think that if Julia requires 300ms to run "hello" then they should make the "hello" benchmark into their primary optimization target. Fortunately, it's easy to make "hello" run fast: just tier your JIT. This can be done with LLVM just fine, see https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ Although we have since moved to using B3, LLVM was absolutely amazing for us, for the same reasons why it's amazing for Julia: you get tons of optimizations for a lot of targets with little effort. But you have to harness this power the right way. You can't fire up the full LLVM pipeline for code that runs once! It's a waste to unleash so many optimizations only to immediately throw away the result.

I had also the feeling the "benchmark" is a joke. Even if it takes some seconds to print hello, speed is only important when you run big programs with million of operations.

pizlonator 10y ago

If it takes a system 300ms to compile a function that prints "hello" then imagine how long it would take to compile a function that does something more sophisticated. The "hello" benchmark is a fantastic benchmark for production-strength JIT-based runtimes because it tells you how long it takes for your system to warm up enough to be able to print something to the screen. You don't want this to be long. I prefer for "hello" to take about 10ms or less. In WebKit we have a few benchmarks that are the moral equivalent of "hello" and these are some of our most important benchmarks. The reason why warm-up time is so insanely important is that any programming language, no matter what the domain is, will be used for the following things eventually: - Tests. - Data definitions in your language (moral equivalent of JSONP - every language/runtime has to deal with something like this). - Initialization. All of these things comprise run-once code that tends to be large. All of these things are important, even in math languages. The glorious "hello" benchmark is an excellent proxy for the performance of these kinds of code. So, I think that if Julia requires 300ms to run "hello" then they should make the "hello" benchmark into their primary optimization target. Fortunately, it's easy to make "hello" run fast: just tier your JIT. This can be done with LLVM just fine, see https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ https://webkit.org/blog/3362/introducing-the-webkit-ftl-jit/ Although we have since moved to using B3, LLVM was absolutely amazing for us, for the same reasons why it's amazing for Julia: you get tons of optimizations for a lot of targets with little effort. But you have to harness this power the right way. You can't fire up the full LLVM pipeline for code that runs once! It's a waste to unleash so many optimizations only to immediately throw away the result.