16 ms·

Garbage collection can cost money to a firm. Working a few years ago in a large investment firm, lots of the code used to trade via algos, make markets, index a

Garbage collection can cost money to a firm. Working a few years ago in a large investment firm, lots of the code used to trade via algos, make markets, index arb, was all written in C++. Deterministic, performant, reliable - it just worked as planned and expected. New tech lead comes in, swaggers about, declares all the street now uses Java for their trading code, we should too. I got the desk heads to listen and be wise to the impending issues, and they told him fine - but if the new Java based code had direct impact on PnL, then his budget, ultimately, he would pay for it. Cocksure, he agreed. Despite throwing bucket loads of Java devs at it, spending fortunes on "tuning" consultants, performance suffered, GC did affect at critical trading conditions, and eventually he was exposed and kicked out. The C++ code came back out of retirement, was updated for C++11/14, and still serves them well to this day.

hga 10y ago
Did he try Azul's Zing or their previous Vega hardware solution for pauseless JVM GC? Not free by any means in $$$ or performance, but they should have avoided any pauses due to GC.
- realo 10y ago
  Do you actually mean that I need an expensive, dedicated hardware accelerator to get Java's performance to match C++? Why not do as they actually did: use C++ ? Sounds simple enough... and much cheaper.
  - egwor 10y ago
    A few things to consider: * it tends to be harder to write good C++ code than Java. For example we think that an average person can write better java code than C++ code because java is simpler to get going with. We also think that the average developer makes fewer mistakes in java than C++ (because C++ allows you to do 'crazy things!') * that means that you need (on average) a better developer for C++, and that tends to cost more * java tooling is exceptional (probably the best there is?) * making small changes and releasing is much quicker in Java; builds tend to be shorter in duration for example * once you have the framework for your system, the differentiator is then the business benefits delivered. If you can push small changes out very quickly you're getting rewards quicker and you can experiment more * once you get down to certain performance points, it becomes similar writing java code as writing C++ code, but the JVM is also doing various work that you can't control so much. That can then inhibit you. Other things: * Azul isn't just h/w development it's also now in kernel plugin form iirc * Have a look at Aeron (https://github.com/real-logic/Aeron https://github.com/real-logic/Aeron) and I think the C++ and Java versions are fairly similar performance and they're designed by super with decent (industry renowned) developers writing it * A lot of people write high performance code making extensive use of templates. That's a totally different way of programming than usual C++
    - Retric 10y ago
      Boring C++ is often faster than boring Java code. It's safer to write hacky Java, but code reviews should not let hacky code in. C++ costs more, but that's often a non issue for a trading floor. Security is much simpler in the Java world, but also possible in boring C++ code. Making small changes is really more a question of your code than anything else IMO. There is a lot of crazy C++ and Java code out there. Now for basic CRUD apps Java is a clear win. For high preformance trading that's an open issue. But, IMO there are much better languages to use.
    - Roboprog 10y ago
      What about a third way? I have not used the GNU version of Ada or Pascal, but you get C/++ style allocation choices (static, stack, heap) without all of the safeties turned off. Having bounds checks (with line number error reporting, rather than secondary, tertiary... damage followed by a core dump), and the use of non-null references (as well as pointers for the initial allocation from the heap) goes a long way towards eliminating much of the time wasting bullshit that C/++ brings into your life. C is good for portable assembler. Otherwise, I want something less wacky than C++ to be "effective" with at a medium low level, and something higher level than Java for most other apps.
      - threeseed 10y ago
        Where are you going to get a team of Ada/Pascal developers from ? And tools to support them ?
        Roboprog 10y ago
        And Enterprise Management has spoken! Ignoring the fantasy that I would get to use something higher level (than Java) for most work... Not everybody panics if they have to write code outside of Eclipse or Visual Studio. Once upon a time, there was an expectation that programmers knew, or could learn, more than one language. Sorry about the rude response, but I'm really sick of the dumbing down of everything for (counterproductive) business or "risk management" purposes. To paraphrase, "average performance" in this industry is pretty poor.
  - loopbit 10y ago
    No, in many cases the JVM performance will match and sometimes even outperform a C++ implementation in terms of performance and I've seen a few examples of that, although math-heavy operations is usually not one of them. In terms of size and memory usage, a C++ implementation will almost always win. The problem with Java is that it's not as deterministic in terms of performance as C++, so in areas where that is a must, it has to be taken into account. Even after saying that, the GC can, and indeed should, be tweaked and optimized. The JVM gives you more tweaking options and tools to optimize the garbage collection than any other platform I've worked with, but most developers don't know about them or don't care. In cases like the original comment mentioned, it may have been better to continue using the C++ implementation, or it might have been useful to spend a bit less in consultants and developers and a bit on Azul's Zing (and maybe a few people that understood the platform). It might have saved a lot in the long term and indeed there are plenty of algorithmic trading solutions written in Java.
  - sievebrain 10y ago
    Azul isn't THAT expensive. These days it's just x86 software anyway. Financial trading firms are actually one of their top customers, if I understand correctly, and Oracle is developing features for low latency work partly driven by the demands of algorithmic trading shops.
    - hga 10y ago
      OK, but it's more expensive that "free", and they didn't previously list their prices, but now I see at the excellent https://www.azul.com/products/zing/zinqfaq/ https://www.azul.com/products/zing/zinqfaq/ Zing is priced on a subscription basis per server.... The annualized subscription price for Zing per physical server ranges from $3500 (for several hundred servers) to $8000. Higher volumes and longer subscription terms will reduce the per-server price for Zing. Pricing for virtual servers is also available upon request. Since you're likely to be running it on a server with a hardware cost approximately the same or larger, next time I'll only use two dollar signs ($$) ^_^. And I've indeed heard it's popular with financial trading firms for the obvious reasons. Up to 1 TiB heap, no problem, and no 1,024 second pause which other collectors require for a full GC.
      - pjmlp 10y ago
        C++ is just free, because you happen to have clang and gcc to choose from among a pool of otherwise commercial compilers. I remember when all C++ compilers worth using were commercial.
lordnacho 10y ago
My experience with trading systems led largely to the same conclusions. You CAN write GC code that does very little collection, but you end up being allocation sensitive anyway, so you may as well write c++ code. For instance I wrote some .NET where instead of logging with strings, I'd use StringBuilder and try to get some reuse. Problem is SB wasn't written for that, and will allocate again if your new string gets too big. In the end it ended up being a pool of stringbuilders of various lengths. C++ would not have had this issue. The problem with GC tends to be something alluded to in the article. When trading is brisk, a lot of data is coming in. So memory pressure is higher, leading to more GC. That's exactly when you don't want it. And debugging managed code I always find is more complicated than debugging C++. You need a whole suite of memory debuggers, all with their own idioms, and then it only really gives you a hint. With C++, you can valgrind et al where you can override new() and put in some accounting code. And it's normally quite clear where you allocated. You can also just tell the OS you'll deal with memory yourself. Just get a big piece, and keep track of what's what using various arenas and such constructs. That way you're not even exposed to the variance in the OS allocation time (time to find space for a new object).
- nulltype 10y ago
  I'm not saying GC is a great idea but c++ doesn't necessarily solve your allocation problem: https://groups.google.com/a/chromium.org/forum/m/#!msg/chromium-dev/EUqoIz2iFU4/kPZ5ZK0K3gEJ https://groups.google.com/a/chromium.org/forum/m/#!msg/chrom... It seems like high performance systems should just be coded like microcontrollers, no OS or minimal embedded real time OS and you manage all your memory yourself.
  - lordnacho 10y ago
    Yes that's right. Essentially you need complete control over what happens, without having to pass control back to the OS. Of course just using c++ is not a solution. There's a bunch of guidelines you need to stick to, especially with string.
nvarsj 10y ago
Quite a few of the top HFT firms use Java, almost exclusively. The trick is to segment your critical path code (whatever is executing the actual trades and is highly susceptible to delays) from any business logic. Then you can focus on making critical path components fast - disable GC completely, keep object allocations way down, audit every line of code, etc. With such a setup you can even do better than typical C++ because you avoid virtually all object allocation/deallocation costs that C++ has. Java object allocation is dirt cheap, it just hurts when you GC, which you can basically avoid or schedule for out of hours. If you want better I would not bother with C++ at all personally, but use C / assembly or even look at FPGA type setups.
- jrcii 10y ago
  I was going to say for people who are going to lengths like kernel bypass networking to get these trades out faster I wouldn't be surprised if the key stuff is in assembly.
  - ryao 10y ago
    I would be. Manually written assembly is not necessary better than what a compiler would generate and with super scalar pipelined architectures, seemingly better assembly can turn out to execute in the same amount of time. The things that matter are algorithmic.
    - slededit 10y ago
      I think you assume compilers are doing more work than they actually are. Processor models in the compiler are generally fairly simplistic and even ICC can easily be beat by hand tuned assembly. Where compilers shine is allowing broad process-wide optimization.
- viraptor 10y ago
  This sounds like simulating arena allocation in Java by disabling GC temporarily. Except it's worse than arena allocator, because it's still mixed up in a general heap and costs more at collection. Not sure I buy the argument that this is cheaper than C++. After all, in C++ you could just drop the whole arena instead, so you get cheap alloc + super-cheap free.
_ph_ 10y ago
A program that segfaults at the wrong moment can ruin your day. It is a tradeoff. It boils down that bad memory handling produces bad results. In C++ these are memory leaks or dereferencing invalid pointers, in GC languages, you pay with performance impacts. And, very occasionally one technology is completely unsuited for a given task. But that should not lead to such blanket statements.
- IshKebab 10y ago
  Well Java can crash and slow down due to GC pauses. But at least you get a nice stack trace.
  - _ph_ 10y ago
    You could call Java a very versatile language :)
saosebastiao 10y ago
What I find sad is that Java garbage collection isn't better than manual memory management, given the vast investment in the jvm. There are plenty of ways in which garbage collection can be faster, but production software is still in the stone ages when compared to research from 10 years ago. For example, any scenario where Rust doesn't even need a lifetime annotation is a scenario where a garbage collecting language compiler still could manually manage collection at almost zero cost. It could, very simply I might add, make inferences of the most efficient way to collect block scoped objects...the vast majority of which end up getting collected in the short lifetime pool. Long lifetime objects might be more efficiently reference counted, so why can't the compiler decide that singletons should be reference counted, or even statically allocated instead of pinged during every large collection? And why can't we have an actual disposable interface that actually performs a manual collection? For a language that is as memory heavy as Java is, with as much boxing that it does, and the high performance demands of its user base, it blows my mind that their best idea for improvement in garbage collection in the last decade was to naively go after more concurrency. I realize that is what everybody thought the future was for everything, but something something Amadahl's law. There are so many better opportunities that involve allocating less, destroying without a mark stage, etc.
- pcwalton 10y ago
  What you're describing is escape analysis and the resulting SROA. These are things that the HotSpot VM does in fact do.
  - saosebastiao 10y ago
    That is news to me (thanks for the clarification), but unless I'm misunderstanding you, escape analysis is only for block scoped allocations. It doesn't perform any optimizations for large and/or long lived objects so they can bypass expensive large heap mark stages, and it doesn't allow for deterministic destructor interfaces. As far as I can tell, the only industrial GC language out there attempting to do this sort of thing is Swift.
    - 10y ago
hntw1 10y ago
This sounds like a hohum IT department. Finance is software and has been for a while. Competent players in this field wouldn't base entire implementation strategy based on one expert view. More likely, they would have more substantial trade off discussions. I would this shop had much bigger problems than choice of stack.
gozur88 10y ago
There's a difference between using a tool that's not up to the job and employing people who aren't up to the job. Java is just fine for this kind of application.

hga 10y ago

Did he try Azul's Zing or their previous Vega hardware solution for pauseless JVM GC? Not free by any means in $$$ or performance, but they should have avoided any pauses due to GC.

realo 10y ago
Do you actually mean that I need an expensive, dedicated hardware accelerator to get Java's performance to match C++? Why not do as they actually did: use C++ ? Sounds simple enough... and much cheaper.
- egwor 10y ago
  A few things to consider: * it tends to be harder to write good C++ code than Java. For example we think that an average person can write better java code than C++ code because java is simpler to get going with. We also think that the average developer makes fewer mistakes in java than C++ (because C++ allows you to do 'crazy things!') * that means that you need (on average) a better developer for C++, and that tends to cost more * java tooling is exceptional (probably the best there is?) * making small changes and releasing is much quicker in Java; builds tend to be shorter in duration for example * once you have the framework for your system, the differentiator is then the business benefits delivered. If you can push small changes out very quickly you're getting rewards quicker and you can experiment more * once you get down to certain performance points, it becomes similar writing java code as writing C++ code, but the JVM is also doing various work that you can't control so much. That can then inhibit you. Other things: * Azul isn't just h/w development it's also now in kernel plugin form iirc * Have a look at Aeron (https://github.com/real-logic/Aeron https://github.com/real-logic/Aeron) and I think the C++ and Java versions are fairly similar performance and they're designed by super with decent (industry renowned) developers writing it * A lot of people write high performance code making extensive use of templates. That's a totally different way of programming than usual C++
  - Retric 10y ago
    Boring C++ is often faster than boring Java code. It's safer to write hacky Java, but code reviews should not let hacky code in. C++ costs more, but that's often a non issue for a trading floor. Security is much simpler in the Java world, but also possible in boring C++ code. Making small changes is really more a question of your code than anything else IMO. There is a lot of crazy C++ and Java code out there. Now for basic CRUD apps Java is a clear win. For high preformance trading that's an open issue. But, IMO there are much better languages to use.
  - Roboprog 10y ago
    What about a third way? I have not used the GNU version of Ada or Pascal, but you get C/++ style allocation choices (static, stack, heap) without all of the safeties turned off. Having bounds checks (with line number error reporting, rather than secondary, tertiary... damage followed by a core dump), and the use of non-null references (as well as pointers for the initial allocation from the heap) goes a long way towards eliminating much of the time wasting bullshit that C/++ brings into your life. C is good for portable assembler. Otherwise, I want something less wacky than C++ to be "effective" with at a medium low level, and something higher level than Java for most other apps.
    - threeseed 10y ago
      Where are you going to get a team of Ada/Pascal developers from ? And tools to support them ?
      - Roboprog 10y ago
        And Enterprise Management has spoken! Ignoring the fantasy that I would get to use something higher level (than Java) for most work... Not everybody panics if they have to write code outside of Eclipse or Visual Studio. Once upon a time, there was an expectation that programmers knew, or could learn, more than one language. Sorry about the rude response, but I'm really sick of the dumbing down of everything for (counterproductive) business or "risk management" purposes. To paraphrase, "average performance" in this industry is pretty poor.
- loopbit 10y ago
  No, in many cases the JVM performance will match and sometimes even outperform a C++ implementation in terms of performance and I've seen a few examples of that, although math-heavy operations is usually not one of them. In terms of size and memory usage, a C++ implementation will almost always win. The problem with Java is that it's not as deterministic in terms of performance as C++, so in areas where that is a must, it has to be taken into account. Even after saying that, the GC can, and indeed should, be tweaked and optimized. The JVM gives you more tweaking options and tools to optimize the garbage collection than any other platform I've worked with, but most developers don't know about them or don't care. In cases like the original comment mentioned, it may have been better to continue using the C++ implementation, or it might have been useful to spend a bit less in consultants and developers and a bit on Azul's Zing (and maybe a few people that understood the platform). It might have saved a lot in the long term and indeed there are plenty of algorithmic trading solutions written in Java.
- sievebrain 10y ago
  Azul isn't THAT expensive. These days it's just x86 software anyway. Financial trading firms are actually one of their top customers, if I understand correctly, and Oracle is developing features for low latency work partly driven by the demands of algorithmic trading shops.
  - hga 10y ago
    OK, but it's more expensive that "free", and they didn't previously list their prices, but now I see at the excellent https://www.azul.com/products/zing/zinqfaq/ https://www.azul.com/products/zing/zinqfaq/ Zing is priced on a subscription basis per server.... The annualized subscription price for Zing per physical server ranges from $3500 (for several hundred servers) to $8000. Higher volumes and longer subscription terms will reduce the per-server price for Zing. Pricing for virtual servers is also available upon request. Since you're likely to be running it on a server with a hardware cost approximately the same or larger, next time I'll only use two dollar signs ($$) ^_^. And I've indeed heard it's popular with financial trading firms for the obvious reasons. Up to 1 TiB heap, no problem, and no 1,024 second pause which other collectors require for a full GC.
    - pjmlp 10y ago
      C++ is just free, because you happen to have clang and gcc to choose from among a pool of otherwise commercial compilers. I remember when all C++ compilers worth using were commercial.

lordnacho 10y ago

My experience with trading systems led largely to the same conclusions. You CAN write GC code that does very little collection, but you end up being allocation sensitive anyway, so you may as well write c++ code. For instance I wrote some .NET where instead of logging with strings, I'd use StringBuilder and try to get some reuse. Problem is SB wasn't written for that, and will allocate again if your new string gets too big. In the end it ended up being a pool of stringbuilders of various lengths. C++ would not have had this issue. The problem with GC tends to be something alluded to in the article. When trading is brisk, a lot of data is coming in. So memory pressure is higher, leading to more GC. That's exactly when you don't want it. And debugging managed code I always find is more complicated than debugging C++. You need a whole suite of memory debuggers, all with their own idioms, and then it only really gives you a hint. With C++, you can valgrind et al where you can override new() and put in some accounting code. And it's normally quite clear where you allocated. You can also just tell the OS you'll deal with memory yourself. Just get a big piece, and keep track of what's what using various arenas and such constructs. That way you're not even exposed to the variance in the OS allocation time (time to find space for a new object).

nulltype 10y ago
I'm not saying GC is a great idea but c++ doesn't necessarily solve your allocation problem: https://groups.google.com/a/chromium.org/forum/m/#!msg/chromium-dev/EUqoIz2iFU4/kPZ5ZK0K3gEJ https://groups.google.com/a/chromium.org/forum/m/#!msg/chrom... It seems like high performance systems should just be coded like microcontrollers, no OS or minimal embedded real time OS and you manage all your memory yourself.
- lordnacho 10y ago
  Yes that's right. Essentially you need complete control over what happens, without having to pass control back to the OS. Of course just using c++ is not a solution. There's a bunch of guidelines you need to stick to, especially with string.

nvarsj 10y ago

Quite a few of the top HFT firms use Java, almost exclusively. The trick is to segment your critical path code (whatever is executing the actual trades and is highly susceptible to delays) from any business logic. Then you can focus on making critical path components fast - disable GC completely, keep object allocations way down, audit every line of code, etc. With such a setup you can even do better than typical C++ because you avoid virtually all object allocation/deallocation costs that C++ has. Java object allocation is dirt cheap, it just hurts when you GC, which you can basically avoid or schedule for out of hours. If you want better I would not bother with C++ at all personally, but use C / assembly or even look at FPGA type setups.

jrcii 10y ago
I was going to say for people who are going to lengths like kernel bypass networking to get these trades out faster I wouldn't be surprised if the key stuff is in assembly.
- ryao 10y ago
  I would be. Manually written assembly is not necessary better than what a compiler would generate and with super scalar pipelined architectures, seemingly better assembly can turn out to execute in the same amount of time. The things that matter are algorithmic.
  - slededit 10y ago
    I think you assume compilers are doing more work than they actually are. Processor models in the compiler are generally fairly simplistic and even ICC can easily be beat by hand tuned assembly. Where compilers shine is allowing broad process-wide optimization.
viraptor 10y ago
This sounds like simulating arena allocation in Java by disabling GC temporarily. Except it's worse than arena allocator, because it's still mixed up in a general heap and costs more at collection. Not sure I buy the argument that this is cheaper than C++. After all, in C++ you could just drop the whole arena instead, so you get cheap alloc + super-cheap free.

_ph_ 10y ago

A program that segfaults at the wrong moment can ruin your day. It is a tradeoff. It boils down that bad memory handling produces bad results. In C++ these are memory leaks or dereferencing invalid pointers, in GC languages, you pay with performance impacts. And, very occasionally one technology is completely unsuited for a given task. But that should not lead to such blanket statements.

IshKebab 10y ago
Well Java can crash and slow down due to GC pauses. But at least you get a nice stack trace.
- _ph_ 10y ago
  You could call Java a very versatile language :)

saosebastiao 10y ago

What I find sad is that Java garbage collection isn't better than manual memory management, given the vast investment in the jvm. There are plenty of ways in which garbage collection can be faster, but production software is still in the stone ages when compared to research from 10 years ago. For example, any scenario where Rust doesn't even need a lifetime annotation is a scenario where a garbage collecting language compiler still could manually manage collection at almost zero cost. It could, very simply I might add, make inferences of the most efficient way to collect block scoped objects...the vast majority of which end up getting collected in the short lifetime pool. Long lifetime objects might be more efficiently reference counted, so why can't the compiler decide that singletons should be reference counted, or even statically allocated instead of pinged during every large collection? And why can't we have an actual disposable interface that actually performs a manual collection? For a language that is as memory heavy as Java is, with as much boxing that it does, and the high performance demands of its user base, it blows my mind that their best idea for improvement in garbage collection in the last decade was to naively go after more concurrency. I realize that is what everybody thought the future was for everything, but something something Amadahl's law. There are so many better opportunities that involve allocating less, destroying without a mark stage, etc.

pcwalton 10y ago
What you're describing is escape analysis and the resulting SROA. These are things that the HotSpot VM does in fact do.
- saosebastiao 10y ago
  That is news to me (thanks for the clarification), but unless I'm misunderstanding you, escape analysis is only for block scoped allocations. It doesn't perform any optimizations for large and/or long lived objects so they can bypass expensive large heap mark stages, and it doesn't allow for deterministic destructor interfaces. As far as I can tell, the only industrial GC language out there attempting to do this sort of thing is Swift.
  - 10y ago

hntw1 10y ago

This sounds like a hohum IT department. Finance is software and has been for a while. Competent players in this field wouldn't base entire implementation strategy based on one expert view. More likely, they would have more substantial trade off discussions. I would this shop had much bigger problems than choice of stack.

gozur88 10y ago

There's a difference between using a tool that's not up to the job and employing people who aren't up to the job. Java is just fine for this kind of application.