5 ms·

Garbage collection. A common pattern in Java based systems is fan-out based processing. One immutable message is created, sent to multiple processors, who then

Garbage collection. A common pattern in Java based systems is fan-out based processing. One immutable message is created, sent to multiple processors, who then combine signals to make a decision. Tracking the memory here would slow you down. But usually the JVM handles this just fine, and kills those objects in first generation. Also, Java has a fantastic standard library, probably the best concurrency system, and the JIT can actually make code faster than C++. It doesn't work for extreme low latency people - those guys write custom code for FPGA - but there is a healthy chunk of HFT for whom Java is a great choice.

easytiger 10y ago
> and the JIT can actually make code faster than C++. How? Demonstrate with real examples
- lucozade 10y ago
  > the JIT can actually make code faster than C++ > How? A JIT compiler can sometimes beat an AOT compiler because it has more information. For example, it is entirely feasible for a JIT to heavily optimise a fast path even if the optimised code wouldn't be correct for all cases that the source could be called for. If the JIT detects an uncommon case it can just fall back to the interpreted code. An AOT compiler will forego optimisations if it can't be sure that it will produce correct code. For example, C++ was generally considered slower than Fortran until restrict was added as the compiler had to be more conservative. However, restrict is, well, restrictive. Conceptually, a JIT could work around this if it had a function that didn't have restrict arguments but was usually called on distinct memory. It could hold an optimised path that assumed restrict but fall back if it detected otherwise. Now, some of this benefit can be had in an AOT compiler with profile guided optimisations. But usually, the AOT compiler will still tend to the conservative to balance aggression with code bloat.
  - gpderetta 10y ago
    It principle yes, but in practice? You see a trend with interpreted, then JITed languages eventually getting an AOT compiler, but I can't recall any natively compiled language getting a JIT for performance reasons (LLVM was supposed to allow that, but the JIT hasn't seen much love)
  - jerven 10y ago
    Current Azul Zing. Is trending into a hybrid, storing previously compiled machine code to use upon restart if byte code matches. I also think that the JVM is the only JIT that has had close to same number of resources pushed at it as some of the Fortran/C/C++ compilers. But to differing markets the JVM focused a lot less on easy to benchmark numeric and actually looks at other types of code. Then of course there is the hybrid JVM/LLVM using Graal/Sulong [1]. Which hopes to better than JIT or AOT apart. [1] https://github.com/graalvm/sulong https://github.com/graalvm/sulong
- Cieplak 10y ago
  http://stackoverflow.com/questions/1878696/why-is-java-faster-when-using-a-jit-vs-compiling-to-machine-code http://stackoverflow.com/questions/1878696/why-is-java-faste...
- yummyfajitas 10y ago
  The jit can optimize away virtual function calls. ParentClass g = ....might be a subclass... for (int i=0;i<100000;i++) { g.func(i); } The JIT can optimize away the vtable lookups to find func, and sometimes inline the code. Ok, maybe you could do this in C++ too. ParentClass[] g = ....might be a subclass... for (int i=0;i<100000;i++) { g[i].func(i); } Suppose 99% of g's are the same class. The JIT can optimize away most of the virtual function lookups (particularly if this is one of your hotspots). I.e. the code becomes: ParentClass[] g = ....might be a subclass... for (int i=0;i<100000;i++) { if (g.class == COMMONCLASS) { inlined_func(i); else { g[i].func(i); } } In the common case, a simple pointer comparison + inlined code is a lot faster than a function dereferene.
  - richardwhiuk 10y ago
    You could, I guess, annotate C++ child classes as likely, and then perform this optimization. That'd be an extension to the standard of course.
  - vitalyd 10y ago
    Devirtualization is mostly an issue for Java since everything is virtual by default and the language doesn't have support for compile time monomorphization. While C++ code does use virtuals, it's nowhere near the amount as Java - there are language constructs to avoid that and move the dispatch selection to compile time.
  - ahh 10y ago
    GCC in fact performs this optimization.
    - yummyfajitas 10y ago
      GCC can get runtime information at compile time? That's a truly advanced compiler.
      - ahh 10y ago
        It can in fact make predictions in a number of circumstances. The most common case is FDO (https://gcc.gnu.org/wiki/AutoFDO https://gcc.gnu.org/wiki/AutoFDO) - with FDO coverage GCC can easily demonstrate a particular vcall is ~always a particular type and emit exactly the code you describe. (To be clear, this isn't just in theory, but is actually happening in a wide variety of binaries I use.) This is perhaps not fully "ahead of time", granted, but it's extremely easy to deploy and highly effective, and entirely accessible to C++.
- Arnt 10y ago
  This mixes two questions: Can java be compiled faster than C++, and can a JIT compiler outperform an AOT compiler. Both java and C++ have problems, C++'s aliasing restrictions are nasty and I'd be surprised if anything java has is as bad. The second is more dubious... sure, a JIT compiler has information if it spends RAM and cycles on collecting that, but it also has to run quickly and fit in the runtime environment, and an AOT compiler can run arbitrarily slowly, use a whole rackful of servers, and can use PGO without incurring any profile collection costs at (normal) runtime.
  - vitalyd 10y ago
    Tiered JITs are meant to allow slower and more aggressive optimizations to be done on truly hot code. However, you're right in that they still cannot spend as much time or resources as an AOT compiler.
    - sa_su_ke_75 10y ago
      The jit must be rerunned every time the process is loaded. In image based language like smalltalk the jit state can be saved in the image with performance optimizations. So next time the image is reloaded the jit status is hot.
      - vitalyd 10y ago
        Yes, Azul has a similar feature (ReadyNow). This is nontrivial because lots of optimizations depend on class load ordering and runtime profile information.
- _pmf_ 10y ago
  The JIT has access to the actual run time class hierarchy and can devirtualize method calls. In C++, if a method is virtual, the system can never decide "oh, this method has only one implementation, so let's always call that" because it must statically assume multiple implementations to be present.
_pmf_ 10y ago
> Tracking the memory here would slow you down. Meh. A lot of systems use specialized memory pools for such use cases: - allocate memory pages from OS - allocate objects via this pool - release the full pool when done (i.e. just unmap t page, which essentially has no cost at all) If the task at hand is known to have an upper bound of same sized objects, this basically reduces management overhead to maintaining a single pointer (and you can work with guard pages to just catch the segfault when trying to access out-of-bound memory, which is not that far fetched because it is actually what some JIT'd Java code does to optimize out null checks [0]). [0] http://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/ http://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/
vitalyd 10y ago
The fantastic standard library mostly goes away because it allocates. It's possible to write Java code that doesn't allocate in steady state, but the coding style becomes terrible (e.g. overuse of primitives, mutable wrappers, manually flattened data structures, etc). There's also the issue that even without GC running you pay the cost of card marking (GC store barriers) on every reference write. There's unpredictability due to deoptimizations occurring due to type/branch profile changes, safepoints occurring due to housekeeping, etc. It's unclear whether that style of Java coding is actually a net win over using languages with better performance model.
- yummyfajitas 10y ago
  No, the GC is a net win from the perspective of code development. The JIT is just one of the things that makes Java not as slow as you'd expect. As I said, the JVM is an acceptable platform for the slower HFT. That's the kind where a clever predictive strategy matters (maybe with lead time of seconds) and you'll get more money from accurately predicting the future than from shaving off 250us. Make no mistake - you'll still make money shaving off 250us, but not so much that you want to be bogged down structuring your code the C++ "if we structure it right we won't leak things" way.
  - vitalyd 10y ago
    You should've made it explicit then that you're referring to slow HFT -- the post I was replying to drew no such distinction apart from saying the "extreme end" uses FPGAs. Obviously if young gen GC pauses aren't an issue, then there's nothing to talk about here but then I'd argue that's not really HFT, although I know the term is quite vague, and is no different than other types of systems. There are other issues with GC and garbage allocations, such as d-cache pollution, but I suppose no need to really discuss them given the type of system you're discussing. I know you were throwing 250us out there as a pseudo example, but that's actually a very long time even outside of UHFT/MM. Also don't forget that your trading daemons will be under a fire hose consuming marketdata, so beyond being able to tick-to-trade quickly, you need to be able to consume that stream without building up a substantial backlog (or worse, OOM or enter permanent gapping).

easytiger 10y ago

> and the JIT can actually make code faster than C++. How? Demonstrate with real examples

lucozade 10y ago
> the JIT can actually make code faster than C++ > How? A JIT compiler can sometimes beat an AOT compiler because it has more information. For example, it is entirely feasible for a JIT to heavily optimise a fast path even if the optimised code wouldn't be correct for all cases that the source could be called for. If the JIT detects an uncommon case it can just fall back to the interpreted code. An AOT compiler will forego optimisations if it can't be sure that it will produce correct code. For example, C++ was generally considered slower than Fortran until restrict was added as the compiler had to be more conservative. However, restrict is, well, restrictive. Conceptually, a JIT could work around this if it had a function that didn't have restrict arguments but was usually called on distinct memory. It could hold an optimised path that assumed restrict but fall back if it detected otherwise. Now, some of this benefit can be had in an AOT compiler with profile guided optimisations. But usually, the AOT compiler will still tend to the conservative to balance aggression with code bloat.
- gpderetta 10y ago
  It principle yes, but in practice? You see a trend with interpreted, then JITed languages eventually getting an AOT compiler, but I can't recall any natively compiled language getting a JIT for performance reasons (LLVM was supposed to allow that, but the JIT hasn't seen much love)
- jerven 10y ago
  Current Azul Zing. Is trending into a hybrid, storing previously compiled machine code to use upon restart if byte code matches. I also think that the JVM is the only JIT that has had close to same number of resources pushed at it as some of the Fortran/C/C++ compilers. But to differing markets the JVM focused a lot less on easy to benchmark numeric and actually looks at other types of code. Then of course there is the hybrid JVM/LLVM using Graal/Sulong [1]. Which hopes to better than JIT or AOT apart. [1] https://github.com/graalvm/sulong https://github.com/graalvm/sulong
Cieplak 10y ago
http://stackoverflow.com/questions/1878696/why-is-java-faster-when-using-a-jit-vs-compiling-to-machine-code http://stackoverflow.com/questions/1878696/why-is-java-faste...
yummyfajitas 10y ago
The jit can optimize away virtual function calls. ParentClass g = ....might be a subclass... for (int i=0;i<100000;i++) { g.func(i); } The JIT can optimize away the vtable lookups to find func, and sometimes inline the code. Ok, maybe you could do this in C++ too. ParentClass[] g = ....might be a subclass... for (int i=0;i<100000;i++) { g[i].func(i); } Suppose 99% of g's are the same class. The JIT can optimize away most of the virtual function lookups (particularly if this is one of your hotspots). I.e. the code becomes: ParentClass[] g = ....might be a subclass... for (int i=0;i<100000;i++) { if (g.class == COMMONCLASS) { inlined_func(i); else { g[i].func(i); } } In the common case, a simple pointer comparison + inlined code is a lot faster than a function dereferene.
- richardwhiuk 10y ago
  You could, I guess, annotate C++ child classes as likely, and then perform this optimization. That'd be an extension to the standard of course.
- vitalyd 10y ago
  Devirtualization is mostly an issue for Java since everything is virtual by default and the language doesn't have support for compile time monomorphization. While C++ code does use virtuals, it's nowhere near the amount as Java - there are language constructs to avoid that and move the dispatch selection to compile time.
- ahh 10y ago
  GCC in fact performs this optimization.
  - yummyfajitas 10y ago
    GCC can get runtime information at compile time? That's a truly advanced compiler.
    - ahh 10y ago
      It can in fact make predictions in a number of circumstances. The most common case is FDO (https://gcc.gnu.org/wiki/AutoFDO https://gcc.gnu.org/wiki/AutoFDO) - with FDO coverage GCC can easily demonstrate a particular vcall is ~always a particular type and emit exactly the code you describe. (To be clear, this isn't just in theory, but is actually happening in a wide variety of binaries I use.) This is perhaps not fully "ahead of time", granted, but it's extremely easy to deploy and highly effective, and entirely accessible to C++.
Arnt 10y ago
This mixes two questions: Can java be compiled faster than C++, and can a JIT compiler outperform an AOT compiler. Both java and C++ have problems, C++'s aliasing restrictions are nasty and I'd be surprised if anything java has is as bad. The second is more dubious... sure, a JIT compiler has information if it spends RAM and cycles on collecting that, but it also has to run quickly and fit in the runtime environment, and an AOT compiler can run arbitrarily slowly, use a whole rackful of servers, and can use PGO without incurring any profile collection costs at (normal) runtime.
- vitalyd 10y ago
  Tiered JITs are meant to allow slower and more aggressive optimizations to be done on truly hot code. However, you're right in that they still cannot spend as much time or resources as an AOT compiler.
  - sa_su_ke_75 10y ago
    The jit must be rerunned every time the process is loaded. In image based language like smalltalk the jit state can be saved in the image with performance optimizations. So next time the image is reloaded the jit status is hot.
    - vitalyd 10y ago
      Yes, Azul has a similar feature (ReadyNow). This is nontrivial because lots of optimizations depend on class load ordering and runtime profile information.
_pmf_ 10y ago
The JIT has access to the actual run time class hierarchy and can devirtualize method calls. In C++, if a method is virtual, the system can never decide "oh, this method has only one implementation, so let's always call that" because it must statically assume multiple implementations to be present.

_pmf_ 10y ago

> Tracking the memory here would slow you down. Meh. A lot of systems use specialized memory pools for such use cases: - allocate memory pages from OS - allocate objects via this pool - release the full pool when done (i.e. just unmap t page, which essentially has no cost at all) If the task at hand is known to have an upper bound of same sized objects, this basically reduces management overhead to maintaining a single pointer (and you can work with guard pages to just catch the segfault when trying to access out-of-bound memory, which is not that far fetched because it is actually what some JIT'd Java code does to optimize out null checks [0]). [0] http://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/ http://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/

vitalyd 10y ago

The fantastic standard library mostly goes away because it allocates. It's possible to write Java code that doesn't allocate in steady state, but the coding style becomes terrible (e.g. overuse of primitives, mutable wrappers, manually flattened data structures, etc). There's also the issue that even without GC running you pay the cost of card marking (GC store barriers) on every reference write. There's unpredictability due to deoptimizations occurring due to type/branch profile changes, safepoints occurring due to housekeeping, etc. It's unclear whether that style of Java coding is actually a net win over using languages with better performance model.

yummyfajitas 10y ago
No, the GC is a net win from the perspective of code development. The JIT is just one of the things that makes Java not as slow as you'd expect. As I said, the JVM is an acceptable platform for the slower HFT. That's the kind where a clever predictive strategy matters (maybe with lead time of seconds) and you'll get more money from accurately predicting the future than from shaving off 250us. Make no mistake - you'll still make money shaving off 250us, but not so much that you want to be bogged down structuring your code the C++ "if we structure it right we won't leak things" way.
- vitalyd 10y ago
  You should've made it explicit then that you're referring to slow HFT -- the post I was replying to drew no such distinction apart from saying the "extreme end" uses FPGAs. Obviously if young gen GC pauses aren't an issue, then there's nothing to talk about here but then I'd argue that's not really HFT, although I know the term is quite vague, and is no different than other types of systems. There are other issues with GC and garbage allocations, such as d-cache pollution, but I suppose no need to really discuss them given the type of system you're discussing. I know you were throwing 250us out there as a pseudo example, but that's actually a very long time even outside of UHFT/MM. Also don't forget that your trading daemons will be under a fire hose consuming marketdata, so beyond being able to tick-to-trade quickly, you need to be able to consume that stream without building up a substantial backlog (or worse, OOM or enter permanent gapping).