7 ms·

I thought the actual approach taken these days is to disable GC for any JVM based trading systems and use arena based architectures or similar for application w

I thought the actual approach taken these days is to disable GC for any JVM based trading systems and use arena based architectures or similar for application where you need super low latency, like a LOB mirror or execution engine, no?

dehrmann 6y ago
I've heard of people doing that 10-15 years ago. I'm sure it still works, but hopefully the GC situation has also improved.
- simonh 6y ago
  A pause is still a pause though. One system I know of had a solution. Throw in a huge amount of RAM on the server and delay garbage collection until end of day.
  - overkalix 6y ago
    Why waste money hire best engineers when more ram does trick?
  - 2sk21 6y ago
    Interesting but doesn't allocation get slower as the heap increases?
    - masklinn 6y ago
      Normally yes but that’s mostly a factor of heap fragmentation. It the GC never runs the “old” buffers are always full so you never bother going through them.
  - maccard 6y ago
    But a pause that can happen when you want it is acceptable. A 3 second stall might be totally acceptable as long as it happens outside of a trade, and you can guarantee that.
    - simonh 6y ago
      If you can predict 3 seconds in advance if you’re going to need to trade or not, you’d make millions very quickly. In trading terms, that’s basically having a time machine.
      - maccard 6y ago
        You don't need to predict in advance, you just need the capability in the system. If a system takes X time from the beginning of a trade to being ready to process the next trade, with Y of that time being GC, it doesn't matter how long Y is if you can execute the entire trade before all of the GC happens.
        simonh 6y ago
        I used to support a derivatives trading system at a company called Patsystems, so I’ve seen this. Our legacy system could process trade triggers from 4ms to 7ms, but tracing the behaviour of the new system it could do it in 3ms, great. Except every now and then a GC pause would halt order processing for around 50ms. Our customers hit the roof. Trading systems trigger almost all their orders based on detecting market changes, which you cannot predict. If a symbol hits a price that triggers one of your trades you want that trade in the market ASAP every millisecond matters. Randomly add 50ms on to that and your trading system is out of business. The customers will go elsewhere.
        maccard 6y ago
        (Sorry, I missed this over the weekend) - I think the issue in this case is you're selling this as a guaranteed 3ms, going down from 4-7ms. If you have to pause for this 50ms GC once every X trades, then you force the GC to happen _after_ the X-1th trade has completed, but you need to provision an extra 15(?) machines to cover the pause. Otherwise, you're not delivering 3ms trades, you're delivering (3 + GCTime/X) ms. It doesn't matter if GC takes 1ms or 1s, you "simply" need to have the capacity to cover the extra GC time.
  - ska 6y ago
    As I understand it (before my time), this was a practical solution found on original symbolics lisp machines; trigger the GC and go get lunch...
- thu2111 6y ago
  There are open source 'pauseless' (really: ultra low pause) GCs for the JVM now. Azul made good money selling such a thing to the finance industry for a long time. Typical pauses may be far less than a millisecond in these GCs because they do almost everything in parallel.
vgatherps 6y ago
Yeah, basically everything has to be that way. At the super fast speeds you start running into things like: * why won’t the devirtualizer trigger? * these object headers are sure wasting a lot of cache * there’s a lot of forced pointer indirection And you just end up spending vast amounts of time and effort trying to shave off those few microseconds you’re wasting in the JVM. Once you add in all the effort trying to get around the GC, it’s bleak picture for competing with quality C++ without spending far more effort.
- blackrock 6y ago
  Why not just write it in C? And pre-allocate all the data structures, make them global, put them into a queue, and constantly reuse them. No more need to instantiate objects.
  - chrisseaton 6y ago
    Why not just write it in assembly while you’re at it? The answer to both questions is that many people prefer writing in higher-level languages with more safety guarantees.
    - mcqueenjordan 6y ago
      This is a bit of a false equivalency -- there aren't many perf benefits in rewriting a C program in asm. The cost you pay for that rewrite is also much greater. Yes, of course $HIGH_LEVEL_LANG is preferable for many many use cases. In this context, we're discussing "high speed trading systems", for which native implementations are going to be the favorite.
      - chrisseaton 6y ago
        > for which native implementations are going to be the favorite ...but the point of the article is they aren't always the favourite.
    - opportune 6y ago
opportune 6y ago
What is an arena based architecture? A quick google didn't show any results. My assumption is that you mean pre-allocating all the memory you want to use at once, but I'm not sure and I'm curious.
- dan-robertson 6y ago
  Basically yeah. You allocate out of pools or arenas. In a language like C this usually means using a block of memory for objects that can all be freed together, so instead of calling malloc and free for each little object in your critical path (eg from request to response on some server, or a frame in a videogame), you somehow determine a maximum size, allocate that (and write enough bytes to have the OS actually give you the memory), and then allocations are just bumping a pointer into this block. At the end of the request, free the block (or return it to a pool to be used later). Sometimes an arena may instead refer to a region of memory for allocations that are all the same size/type. Because everything is the same size, the data structures for tracking allocations may be simpler. In a language like Java, you basically preallocate an array of objects of the same type with a bunch of fields set to zero, and then have some data structure to give you a fast interface a bit like malloc. Because you don’t do any extra allocation, the GC can be safely turned off.

dehrmann 6y ago

I've heard of people doing that 10-15 years ago. I'm sure it still works, but hopefully the GC situation has also improved.

simonh 6y ago
A pause is still a pause though. One system I know of had a solution. Throw in a huge amount of RAM on the server and delay garbage collection until end of day.
- overkalix 6y ago
  Why waste money hire best engineers when more ram does trick?
- 2sk21 6y ago
  Interesting but doesn't allocation get slower as the heap increases?
  - masklinn 6y ago
    Normally yes but that’s mostly a factor of heap fragmentation. It the GC never runs the “old” buffers are always full so you never bother going through them.
- maccard 6y ago
  But a pause that can happen when you want it is acceptable. A 3 second stall might be totally acceptable as long as it happens outside of a trade, and you can guarantee that.
  - simonh 6y ago
    If you can predict 3 seconds in advance if you’re going to need to trade or not, you’d make millions very quickly. In trading terms, that’s basically having a time machine.
    - maccard 6y ago
      You don't need to predict in advance, you just need the capability in the system. If a system takes X time from the beginning of a trade to being ready to process the next trade, with Y of that time being GC, it doesn't matter how long Y is if you can execute the entire trade before all of the GC happens.
      - simonh 6y ago
        I used to support a derivatives trading system at a company called Patsystems, so I’ve seen this. Our legacy system could process trade triggers from 4ms to 7ms, but tracing the behaviour of the new system it could do it in 3ms, great. Except every now and then a GC pause would halt order processing for around 50ms. Our customers hit the roof. Trading systems trigger almost all their orders based on detecting market changes, which you cannot predict. If a symbol hits a price that triggers one of your trades you want that trade in the market ASAP every millisecond matters. Randomly add 50ms on to that and your trading system is out of business. The customers will go elsewhere.
        maccard 6y ago
        (Sorry, I missed this over the weekend) - I think the issue in this case is you're selling this as a guaranteed 3ms, going down from 4-7ms. If you have to pause for this 50ms GC once every X trades, then you force the GC to happen _after_ the X-1th trade has completed, but you need to provision an extra 15(?) machines to cover the pause. Otherwise, you're not delivering 3ms trades, you're delivering (3 + GCTime/X) ms. It doesn't matter if GC takes 1ms or 1s, you "simply" need to have the capacity to cover the extra GC time.
- ska 6y ago
  As I understand it (before my time), this was a practical solution found on original symbolics lisp machines; trigger the GC and go get lunch...
thu2111 6y ago
There are open source 'pauseless' (really: ultra low pause) GCs for the JVM now. Azul made good money selling such a thing to the finance industry for a long time. Typical pauses may be far less than a millisecond in these GCs because they do almost everything in parallel.

vgatherps 6y ago

Yeah, basically everything has to be that way. At the super fast speeds you start running into things like: * why won’t the devirtualizer trigger? * these object headers are sure wasting a lot of cache * there’s a lot of forced pointer indirection And you just end up spending vast amounts of time and effort trying to shave off those few microseconds you’re wasting in the JVM. Once you add in all the effort trying to get around the GC, it’s bleak picture for competing with quality C++ without spending far more effort.

blackrock 6y ago
Why not just write it in C? And pre-allocate all the data structures, make them global, put them into a queue, and constantly reuse them. No more need to instantiate objects.
- chrisseaton 6y ago
  Why not just write it in assembly while you’re at it? The answer to both questions is that many people prefer writing in higher-level languages with more safety guarantees.
  - mcqueenjordan 6y ago
    This is a bit of a false equivalency -- there aren't many perf benefits in rewriting a C program in asm. The cost you pay for that rewrite is also much greater. Yes, of course $HIGH_LEVEL_LANG is preferable for many many use cases. In this context, we're discussing "high speed trading systems", for which native implementations are going to be the favorite.
    - chrisseaton 6y ago
      > for which native implementations are going to be the favorite ...but the point of the article is they aren't always the favourite.
  - opportune 6y ago

opportune 6y ago

What is an arena based architecture? A quick google didn't show any results. My assumption is that you mean pre-allocating all the memory you want to use at once, but I'm not sure and I'm curious.

dan-robertson 6y ago
Basically yeah. You allocate out of pools or arenas. In a language like C this usually means using a block of memory for objects that can all be freed together, so instead of calling malloc and free for each little object in your critical path (eg from request to response on some server, or a frame in a videogame), you somehow determine a maximum size, allocate that (and write enough bytes to have the OS actually give you the memory), and then allocations are just bumping a pointer into this block. At the end of the request, free the block (or return it to a pool to be used later). Sometimes an arena may instead refer to a region of memory for allocations that are all the same size/type. Because everything is the same size, the data structures for tracking allocations may be simpler. In a language like Java, you basically preallocate an array of objects of the same type with a bunch of fields set to zero, and then have some data structure to give you a fast interface a bit like malloc. Because you don’t do any extra allocation, the GC can be safely turned off.