5 ms·

Superfast asm.js benchmarks of V8 TurboFan

37 points by Pharohbot 12y ago

desdiv 12y ago
How the hell did C++ lose to Javascript on the first benchmark!? And it's not just a little slower either, it's 36% slower than the fastest Javascript implementation.
- stingraycharles 12y ago
  For those interested, here is the C++ code of the first benchmark: https://github.com/h4writer/arewefastyet/blob/master/benchmarks/asmjs-ubench/copy.cpp https://github.com/h4writer/arewefastyet/blob/master/benchma...
  - ridiculous_fish 12y ago
    Oh yeah, I love this stuff. On my Linux box, clang 3.4 is 50% slower than gcc 4.9 on -O3 on that code, for all inputs. The key difference is, unsurprisingly, in the inner block: for (int j = 0; j < 50000; j++)... clang's inner block asm: https://gist.github.com/ridiculousfish/24b16ba5b4e8abbe051b https://gist.github.com/ridiculousfish/24b16ba5b4e8abbe051b gcc's inner block asm: https://gist.github.com/ridiculousfish/bee5217d6ae1f4f34a5d https://gist.github.com/ridiculousfish/bee5217d6ae1f4f34a5d Notice that clang's is much longer. Also notice all of the "# imm = 0xFFFFFFFFFFFFFC00" lines in clang. Those immediate values correspond to inlining the norm() functions: it's how clang is computing x%1024. How do you compute x%1024? It sure would be nice to strength-reduce it to x&1023, but that's always going to produce a non-negative number, which is not what you want if x is negative. That is, -1025 % 1024 ought to be -1, and not 1023. So that strength reduction (x%1024 -> x&1023) is only valid if you can prove that x is non-negative. And that's the first big difference. gcc-4.8 convinces itself that all of the ints involved are non-negative, and so it computes x%1024 as x&1023. clang fails to notice that all ints are non-negative, and so emits the slow path for the mod. Let's replace int with unsigned int, and see what happens: clang's uint asm: https://gist.github.com/ridiculousfish/c35ffcff5963160c9622 https://gist.github.com/ridiculousfish/c35ffcff5963160c9622 This allows the x&1023 strength reduction, but now we hit the second big difference: clang is futzing with xmm registers, perhaps packing a single 'vec' into a register. The codegen just looks silly to me - all those pshufd and punpckldq. Let's disable that noise with -mno-sse: clang's uint asm (no SSE): https://gist.github.com/ridiculousfish/0afc79d458ae30dc6327 https://gist.github.com/ridiculousfish/0afc79d458ae30dc6327 Now clang is totally kicking ass, with fewer instructions than gcc, and about 17% faster. (Of course gcc may in turn be massaged to be even faster yet.) Timings on my i3 NUC with -O3, input 4. Lower is better: gcc w/ signed: 4.78 sec clang w/ signed: 7.11 sec gcc w/ unsigned: 3.83 sec clang w/ unsigned: 7.11 sec clang w/ unsigned no-sse: 3.14 sec In summary, clang is missing a big optimization, and then emitting some dubious SSE code. Fixing those more than doubles its speed. This illustrates the importance of monitoring your codegen for tight loops, and the hazards of micro benchmarks!
    - detrino 12y ago
      I think most of the SSE slowness is because the weird static vec::add that uses pass by value, try changing that to pass by reference.
      - ridiculous_fish 12y ago
        Wow, you're right! That improves the time from 7.11 to 3.94 seconds. I was skeptical because 'add' is inlined, which I expected would eliminate pointless copies, but I guess not!
- vardump 12y ago
  Plain C++ can easily lose to JITted scripting languages. Nothing new or weird in that. Remember, we're comparing native code to native code.
  - accurrent 12y ago
    Not true actually theres always a JIT overhead for the compiler.
    - zamalek 12y ago
      Not only does JITting normally only occur once (something that microbenchmarks take into consideration by "warming up" the method), but if it does occur an Nth time it's because the runtime has acquired profiling information to do a more intelligent optimization pass on the method: far exceeding the quality of assumptions/heuristics made during the first JIT pass. As far as I know Hotspot implements PGO JITting. Static compilers can do profile guided optimization, but this only takes into account the static scenario that you profiled - not the real world scenario that the program is encountering. Your PGO is only as good as your profile. In theory, not only is your statement untrue - but reality can turn out to be the exact opposite.
    - vardump 12y ago
      There's also a significant overhead for not being able to target exactly the CPU model and parameters for the call at compile time. Unless of course doing profile guided optimization just for that CPU model and use case. I've been playing around with an idea for a long time to write a JIT for native code. I think it's possible to speed up most native code by JITting it. Sadly it would take much more time I can afford to spend. The principle is sound, though. It'd be possible to eliminate a lot of computation at runtime. Remove a lot of branches, use CPU model specific instructions when a suitable pattern is detected, etc. Functions could be simplified to constraints, function calls could be dynamically inlined. Spilled stack variables could be allocated in registers. Calling conventions optimized into passing more parameters in the registers. On register starved 32-bit x86, registers for parameters with effectively constant values could be converted into instructions with immediate values, saving registers for actually changing data. There's also no reason why you can't cache previously JITted code into a file.
- detrino 12y ago
  Two things to keep in mind: 1) This program has no negative numbers yet the c++ source uses signed ints while having a lot of modulo calls which are slower than if unsigned int was used 2) This program was benchmarked with clang++ In the following numbers "new program" is replacing signed int with unsigned int and all times are measured with perf stat -r5 <binary> and run on an old intel wolfdale cpu. Original program compiled with clang++ -O3: 1.441151117s Original program compiled with g++ -O3: 0.826723649s New program compiled with clang++ -O3: 1.181305003s New program compiled with g++ -O3: 0.721448016s Edit: Newer program with proper pass by reference vec::add Newer program compiled with clang++ -O3: 0.770128848s
  - vardump 12y ago
    Yeah, same silly things programmers do in real world statically compiled code. Bad choices for something like data type and bad compiler options. A JIT can figure out negative numbers can't happen and optimize accordingly. It might know about the CPU model that didn't even exist when the static code was compiled.
    - detrino 12y ago
      And how do you expect a JIT to "figure out" negative numbers can't happen and optimize accordingly where a static compiler could not?
      - vardump 12y ago
        It can do simple control flow analysis and data flow analysis and generate the constraints out of that. It doesn't even need to be right, if there's a proper guard condition in the generated code.
        detrino 12y ago
        Interestingly, you brought up two things that are typically categorized as static analysis. Keep in mind GCC does figure out the values will never be negative and its run-times destroy all the JIT competitors listed on that page. Also, static compilers can and do generate code with guard conditions.
azakai 12y ago
1. That is on 32-bit, for fairness it's worth mentioning that Safari is slow there because its FTL JIT is 64-bit only. 2. Firefox would also be faster on 64-bit as well (less significantly). However, TurboFan results aren't showing up on the 64-bit bot - not sure if that's an issue with the bot, or if TurboFan doesn't build on 64-bit yet. 3. Those are microbenchmarks, for more realistic workloads click on "apps", which leads to http://arewefastyet.com/#machine=28&view=breakdown&suite=asmjs-apps http://arewefastyet.com/#machine=28&view=breakdown&suite=asm...
- zobzu 12y ago
  Looks more like firefox + ion monkey is superfast on asm.js, not chrome which.. actually reflects reality.
cromwellian 12y ago
Is Turbofan even beta yet? Seems a bit early, it performs worse than crankshaft on some code at the moment.

37 points by Pharohbot 12y ago

How the hell did C++ lose to Javascript on the first benchmark!? And it's not just a little slower either, it's 36% slower than the fastest Javascript implementation.

stingraycharles 12y ago
For those interested, here is the C++ code of the first benchmark: https://github.com/h4writer/arewefastyet/blob/master/benchmarks/asmjs-ubench/copy.cpp https://github.com/h4writer/arewefastyet/blob/master/benchma...
- ridiculous_fish 12y ago
  Oh yeah, I love this stuff. On my Linux box, clang 3.4 is 50% slower than gcc 4.9 on -O3 on that code, for all inputs. The key difference is, unsurprisingly, in the inner block: for (int j = 0; j < 50000; j++)... clang's inner block asm: https://gist.github.com/ridiculousfish/24b16ba5b4e8abbe051b https://gist.github.com/ridiculousfish/24b16ba5b4e8abbe051b gcc's inner block asm: https://gist.github.com/ridiculousfish/bee5217d6ae1f4f34a5d https://gist.github.com/ridiculousfish/bee5217d6ae1f4f34a5d Notice that clang's is much longer. Also notice all of the "# imm = 0xFFFFFFFFFFFFFC00" lines in clang. Those immediate values correspond to inlining the norm() functions: it's how clang is computing x%1024. How do you compute x%1024? It sure would be nice to strength-reduce it to x&1023, but that's always going to produce a non-negative number, which is not what you want if x is negative. That is, -1025 % 1024 ought to be -1, and not 1023. So that strength reduction (x%1024 -> x&1023) is only valid if you can prove that x is non-negative. And that's the first big difference. gcc-4.8 convinces itself that all of the ints involved are non-negative, and so it computes x%1024 as x&1023. clang fails to notice that all ints are non-negative, and so emits the slow path for the mod. Let's replace int with unsigned int, and see what happens: clang's uint asm: https://gist.github.com/ridiculousfish/c35ffcff5963160c9622 https://gist.github.com/ridiculousfish/c35ffcff5963160c9622 This allows the x&1023 strength reduction, but now we hit the second big difference: clang is futzing with xmm registers, perhaps packing a single 'vec' into a register. The codegen just looks silly to me - all those pshufd and punpckldq. Let's disable that noise with -mno-sse: clang's uint asm (no SSE): https://gist.github.com/ridiculousfish/0afc79d458ae30dc6327 https://gist.github.com/ridiculousfish/0afc79d458ae30dc6327 Now clang is totally kicking ass, with fewer instructions than gcc, and about 17% faster. (Of course gcc may in turn be massaged to be even faster yet.) Timings on my i3 NUC with -O3, input 4. Lower is better: gcc w/ signed: 4.78 sec clang w/ signed: 7.11 sec gcc w/ unsigned: 3.83 sec clang w/ unsigned: 7.11 sec clang w/ unsigned no-sse: 3.14 sec In summary, clang is missing a big optimization, and then emitting some dubious SSE code. Fixing those more than doubles its speed. This illustrates the importance of monitoring your codegen for tight loops, and the hazards of micro benchmarks!
  - detrino 12y ago
    I think most of the SSE slowness is because the weird static vec::add that uses pass by value, try changing that to pass by reference.
    - ridiculous_fish 12y ago
      Wow, you're right! That improves the time from 7.11 to 3.94 seconds. I was skeptical because 'add' is inlined, which I expected would eliminate pointless copies, but I guess not!
vardump 12y ago
Plain C++ can easily lose to JITted scripting languages. Nothing new or weird in that. Remember, we're comparing native code to native code.
- accurrent 12y ago
  Not true actually theres always a JIT overhead for the compiler.
  - zamalek 12y ago
    Not only does JITting normally only occur once (something that microbenchmarks take into consideration by "warming up" the method), but if it does occur an Nth time it's because the runtime has acquired profiling information to do a more intelligent optimization pass on the method: far exceeding the quality of assumptions/heuristics made during the first JIT pass. As far as I know Hotspot implements PGO JITting. Static compilers can do profile guided optimization, but this only takes into account the static scenario that you profiled - not the real world scenario that the program is encountering. Your PGO is only as good as your profile. In theory, not only is your statement untrue - but reality can turn out to be the exact opposite.
  - vardump 12y ago
    There's also a significant overhead for not being able to target exactly the CPU model and parameters for the call at compile time. Unless of course doing profile guided optimization just for that CPU model and use case. I've been playing around with an idea for a long time to write a JIT for native code. I think it's possible to speed up most native code by JITting it. Sadly it would take much more time I can afford to spend. The principle is sound, though. It'd be possible to eliminate a lot of computation at runtime. Remove a lot of branches, use CPU model specific instructions when a suitable pattern is detected, etc. Functions could be simplified to constraints, function calls could be dynamically inlined. Spilled stack variables could be allocated in registers. Calling conventions optimized into passing more parameters in the registers. On register starved 32-bit x86, registers for parameters with effectively constant values could be converted into instructions with immediate values, saving registers for actually changing data. There's also no reason why you can't cache previously JITted code into a file.
detrino 12y ago
Two things to keep in mind: 1) This program has no negative numbers yet the c++ source uses signed ints while having a lot of modulo calls which are slower than if unsigned int was used 2) This program was benchmarked with clang++ In the following numbers "new program" is replacing signed int with unsigned int and all times are measured with perf stat -r5 <binary> and run on an old intel wolfdale cpu. Original program compiled with clang++ -O3: 1.441151117s Original program compiled with g++ -O3: 0.826723649s New program compiled with clang++ -O3: 1.181305003s New program compiled with g++ -O3: 0.721448016s Edit: Newer program with proper pass by reference vec::add Newer program compiled with clang++ -O3: 0.770128848s
- vardump 12y ago
  Yeah, same silly things programmers do in real world statically compiled code. Bad choices for something like data type and bad compiler options. A JIT can figure out negative numbers can't happen and optimize accordingly. It might know about the CPU model that didn't even exist when the static code was compiled.
  - detrino 12y ago
    And how do you expect a JIT to "figure out" negative numbers can't happen and optimize accordingly where a static compiler could not?
    - vardump 12y ago
      It can do simple control flow analysis and data flow analysis and generate the constraints out of that. It doesn't even need to be right, if there's a proper guard condition in the generated code.
      - detrino 12y ago
        Interestingly, you brought up two things that are typically categorized as static analysis. Keep in mind GCC does figure out the values will never be negative and its run-times destroy all the JIT competitors listed on that page. Also, static compilers can and do generate code with guard conditions.

azakai 12y ago

1. That is on 32-bit, for fairness it's worth mentioning that Safari is slow there because its FTL JIT is 64-bit only. 2. Firefox would also be faster on 64-bit as well (less significantly). However, TurboFan results aren't showing up on the 64-bit bot - not sure if that's an issue with the bot, or if TurboFan doesn't build on 64-bit yet. 3. Those are microbenchmarks, for more realistic workloads click on "apps", which leads to http://arewefastyet.com/#machine=28&view=breakdown&suite=asmjs-apps http://arewefastyet.com/#machine=28&view=breakdown&suite=asm...

zobzu 12y ago
Looks more like firefox + ion monkey is superfast on asm.js, not chrome which.. actually reflects reality.

cromwellian 12y ago

Is Turbofan even beta yet? Seems a bit early, it performs worse than crankshaft on some code at the moment.