8 ms·
Let me put my perspective on this. As it happens, I have developed one algorithmic, low latency trading system in Common Lisp / ANSI C, and then was asked to r
by lmilcin 6y ago
Let me put my perspective on this.
As it happens, I have developed one algorithmic, low latency trading system in Common Lisp / ANSI C, and then was asked to rewrite it in Java which I did.
It actually traded on Warsaw Stock Exchange and was certified by WSE and was connected directly to it (no intervening software).
Yes, it is possible to do really low latency in Java. My experience is my optimized Java code is about 20-30% slower than what very optimized C code which is absolutely awesome. We are talking real research project funded by one of larger brokerage houses and the application was expected to respond to market signals within single digit microseconds every single time.
The issue with Java isn't that you can't write low latency code. The issue is that you are left almost completely without tools as you can't use almost anything from Java standard library and left scratching your head how to solve even simplest problem like managing memory without something coming and suddenly creating a lot of latency where you don't want it to happen.
Can't use interfaces, can't use collections, can't use exceptions, etc.
You can forget about freely allocating objects, except for very short lived objects you must relegate yourself to managing preallocated pools of objects.
I don't know what is the current state of garbage collection but in our case garbage collection had to be turned off and nothing outside of Eden could be collected during program execution (Java 8).
Writing that kind of optimized code is all about control and Java is all about abstracting it so you don't have to worry. As you see, both goals are at odds.
So I still prefer working low latency in C as it is more natural, native solution for managing problem where you absolutely need to control memory layouts, preallocated object pools, NUMA, compilation of decision trees to machine code, prefetcher, etc.
I have about 10 years combined experience working with C and 15 years working with Java.
- news_to_me 6y agoI'm curious, what role did Common Lisp play in the original version of your trading system?
- lmilcin 6y agoCommon Lisp was doing all the "high level" stuff. No CL code was on the actual critical path but it still was responsible for some critical things that would be difficult otherwise. For example, I had piece of Common Lisp code optimize and compile decision trees to machine code. These were used, for example, to check whether the trade is allowed to go to market. This had to be very optimized as it was running in parallel to constructing the message and had to deliver verdict before the message was actually created to have a chance at actually stopping anything from reaching exchange. This was my solution to have order verification in essentially zero time (the cost was single branch instruction that would only be mispredicted if the verdict was to stop the order). Another piece of Common Lisp generated machine code to parse XDP messages coming from exchange. The spec for these was in XML, Common Lisp macros created extremely optimized machine code that used absolute minimum instructions to parse out the information we were interested in without having to write all that code manually. I actually used concept from Practical Common Lisp book for this (http://www.gigamonkeys.com/book/practical-parsing-binary-files.html http://www.gigamonkeys.com/book/practical-parsing-binary-fil...). Thanks, Peter Seibel. The entire application was essentially a Common Lisp application that used FFI to call a library with optimized ANSI C code. Some Common Lisp code was running in parallel to constantly update things like decision trees that required regular recompiling based on market situation but most was really just to setup everything and give control to the C part. The application did not talk to operating system (Linux) after it started (no syscalls at all) and used full kernel bypass to talk to NIC.
- matheusmoreira 6y ago> did not talk to operating system (Linux) after it started (no syscalls at all) and used full kernel bypass to talk to NIC That's really interesting... How does it work? How can a user space program talk to the NIC directly without using system calls? Shared memory?
- eru 6y agoSome Googling suggests https://blog.cloudflare.com/kernel-bypass/ https://blog.cloudflare.com/kernel-bypass/
- clappski 6y agoA lot of NIC manufactures provide either some hooks into something like DPDK or simple LD_PRELOADable libraries that offload the network syscalls (socket, poll, recv, etc.) into a library that talks to the NIC directly using mmap'd IO I presume.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- bigbubba 6y agoWas a specialized real-time JVM used for this? I know real time isn't necessarily low latency, but it sounds like both are desired properties in this sort of situation.
- junon 6y agoRealtime and low latency are two different properties. Realtime systems have timing concerns and can oftentimes be slower than non-realtime systems.
- lmilcin 6y agoThis is not strictly true. Realtime means processing of at least some of the events and tasks in your application must be completed before a set deadline. For example, coin acceptor that detects what kind of coin you dropped in has x milliseconds to figure out if the coin is to be accepted or not. If it can't do it within the time you get unhappy customer. Realtime systems don't have to be very performant, but they don't have something like 99th percentile latency shooting up. All events must be processed within deadline. Low latency systems are soft realtime PLUS performance. In a low latency trading system you have to respond to market within a deadline to have a chance to do something useful but if you don't normally it doesn't mean lives are lost, metal is bent, etc.
- iphorde 6y agoRealtime requires low-latency. In trading, they require real-time, not low-latency.
- peter_lawrey 6y agoSome people use Azul Zing but we typically use OpenJDK or Oracle JDK.
- rafaelvasco 6y agoYeah. Languages like C# and Java offer you so much facilities and QOL features compared to C and C++, but in practice you have to give up most of those facilities in order to have it perform almost on par with lower level languages; Even then I think it's still acceptable since you still have the tooling, the facilitated debugging, etc. C# for example, is keeping up with this trend of using higher level languages to code things closer to the metal and with more memory management awareness, and offering more and more features to make that kind of code more performant and with less heap allocations, since C# 7.0 or so: Span<T> for example is a revolution; But I use things like stackalloc, ref, fixed, Unsafe.CopyBlockUnaligned, &MemoryMarshal.GetReference etc, on a daily basis to get that extra performance where it matters;
- pjmlp 6y agoThat was exactly what made me pick C++ over C already in 1992, ability to code at all levels in a modern programming language. Nowadays I rather use Java and .NET, which have been picking up on the features that should have been there at v1.0, like the ones you mentioned, but hey better later than never. :) And on the few cases where it isn't enough, I can code a tiny library in C or C++, no need to throw everything away.
- junon 6y agoI'm so happy this is the top comment. I'm saving this as it's well written and highly qualified. Thanks for sharing.
- DC-3 6y agoSo Java is nearly as low-latency as C if you're willing to write your Java as if it were C?
- ByteJockey 6y agoYou're really just paying for the jvm overhead at that point. 20-30% seems in the right ballpark.
- lmilcin 6y agoIt can be done. It is just not very practical, you are better off writing C. The only situation I can think of where Java would be right fit is where you have an application with a lot of logic and just a small part that needs to be highly optimized.
- peter_lawrey 6y agoMost trading systems I work on only need a very small portion to be highly optimised. As long as most of the code is pretty efficient that is fine.
- iphorde 6y agoNo. Java is not low-latency by any stretch. Java advocates would like to think it is, until a Bank, like BoA spends $8 million on Java in 2014, just to have it fail compliance testing. Entire project was scrapped.
- vbezhenar 6y agoSounds like it should be possible to wrap it all into some kind of library and set of guidelines. On a positive side you probably could have much more confidence in Java code, because lots of boring and hard to debug memory errors just not possible (but may be C professionals don't do those errors?).
- lmilcin 6y agoThe cost of environment not making it possible for you to make a mistake is always there, somewhere. I think best hypothetical development environment would allow you to specify the tradeoff and choose whether you want strong typing, fully managed memory or ownership semantics.
- saghm 6y ago> So I still prefer working low latency in C as it is more natural, native solution for managing problem where you absolutely need to control memory layouts, preallocated object pools, NUMA, compilation of decision trees to machine code, prefetcher, etc. Out of curiosity, what was the rationale for writing the system in Java? Were the tradeoffs not obvious until too late, was the choice of language dictated by higher-ups, etc.?
- stingraycharles 6y agoProbably the usual suspect, it’s easier to find engineers that know Java than C/C++.
- RobLach 6y agoAnd it’s a foolish assessment. Finding Java programmers who can navigate all the hoops of low latency Java is more difficult than finding a C programmer who can do the same with C.
- lmilcin 6y agoThere is couple of reasons for this. If you choose Java for your project it usually means you don't care a lot about latency and performance (you care some, but not all that much compared to, for example, linux kernel development). You care more about managing larger codebase, having easy access to whole lot of frameworks and libraries, good development tools and easy development / deployment process. Developers who have experience with Java are inevitably people who have mostly had experience with those kinds of projects of which I would say most are corporate backend systems or Android applications. It really is difficult to find any other projects. On the other C is inflicted pain of having much harder development environment for the size of the project. You only use low level language like C when you have a specific need. Maybe you need to develop Set Top Box software, or video codec, or some device controller, or high performance quant library, that kind of stuff. So it isn't really fault of Java developers to not know the stuff. It is natural result of Java being suited to a different type of projects than C is.
- 6y ago
- Rebelgecko 6y agoOne interesting approach I heard about for doing garbage collection in finance is to get computers with a shitload of RAM and just turn off GC. At the end of the trading day you can reclaim memory by killing the process. Not sure how common that strategy is.
- gbronner 6y agoIt works until it doesn't and you lose a ton of money
- lmilcin 6y agoThat is exactly what I did. It was much less elegant and performant, though, than my initial solution with Common Lisp and C which basically allocated buffers in memory and mapped them with huge pages so that memory access don't ever require switch to kernel for mapping. We also had custom Intel CPUs that are auctioned individually. And cutthrough switches that stream the IP packet immediately when they receive the IP header, before they receive the payload. I have calculated that if the message from exchange was a beam of light it would be delayed by less than a meter.
- KMag 6y ago> would be delayed by less than a meter That's just latency to your NIC, right? 1 meter ~ 3 ns ~ 9 CPU cycles at 3 GHz, less than the integer pipeline depth last I checked.
- lmilcin 6y agoLatency through switch (ie. how much the switch added compared to if the signal traveled unimpeded in vacuum). Decade ago.
- dralley 6y agoIf that's your memory management strategy, then what do you actually gain by using Java? May as well just use C and never free your memory. Lots of memory management issues go away if you never free memory.