8 ms·
Ultra-low-latency, batching and concurrent queue for IPC in Java
- WeaselNo7 2y agoWeird to see this here! I've used CoralBlocks in the low-latency trading domain previously. Highly recommend. The API is kind, they're very responsive, and the latency is exceptional (and comes with all the basics like thread pinning built-in for convenience)
- smarx007 2y agoHow does it compare to LMAX Disruptor if you have any experience with both?
- papercrane 2y agoThey're both similar in design, the main difference is Coral Queue can be used for IPC between JVMs, using a mmap'd file.
- arnold_palmur 2y agoSee https://github.com/real-logic/aeron https://github.com/real-logic/aeron (also from the creator of the disruptor)
- algo_trader 2y ago> CoralBlocks in the low-latency trading domain previously. Yeah, modern JVM is a true miracle and you can be x5 productive (and safe!) compared to C/C++ Do you have any recommendations for a low latency work queue (with in a jvm)? I want to spawn millions of micro-second-tasks per second, to worker cores.. I am on a massive cache CPU so memory latency hasnt raised its ugly head yet EDIT: not LMAX please...
- jffhn 2y ago>any recommendations for a low latency work queue (with in a jvm)? I toyed around the ring buffer pattern a decade ago, creating a unicast one (using CAS on entries, and eventually a logarithmic scan for next readable entry, not to brute-force-scan them all), but I'm not sure that its latency is much better than that of a regular ThreadPoolExecutor (the throughput could be better though). Latency also depends on whether it spins or blocks when waiting for a slot to read or write. If you want to give it a try: https://github.com/jeffhain/jodk/blob/master/src/net/jodk/threading/ringbuffers/UnicastRingBuffer.java https://github.com/jeffhain/jodk/blob/master/src/net/jodk/th...
- pacoverdi 2y agoYou may find https://jctools.github.io/JCTools/ https://jctools.github.io/JCTools/ interesting
- joas_coder 2y agoI think you can take a look at the DiamondQueue, which is a Demultiplexer and a Multiplexer combined so that thread A dispatches a bunch of tasks to a fixed set of worker threads (W1, W2, W3, etc.) and then these worker threads use a Multiplexer to deliver the results to another thread B. Thread A and Thread B can also be the same thread. There is an example here => https://www.coralblocks.com/index.php/the-diamond-queue-demultiplexer-to-set-of-threads-to-multiplexer/ https://www.coralblocks.com/index.php/the-diamond-queue-demu... The DiamondQueue should be soon available for free at the CoralQueue project on GitHub.
- joas_coder 2y agoYou were interested in the diamond queue, so just wanted to say that it is now released in our GitHub => https://github.com/coralblocks/CoralQueue?tab=readme-ov-file#diamond https://github.com/coralblocks/CoralQueue?tab=readme-ov-file...
- bsder 2y agoGiven the documentation says that this is supposedly to be between JVMs, how do they handle the serialize/deserialize?
- kasey_junk 2y agoThey punt on the actual serialization format: https://www.coralblocks.com/index.php/inter-process-communication-with-coralqueue/ https://www.coralblocks.com/index.php/inter-process-communic... In most applications like this you'll see direct byte manipulation to byte buffers because you want to pull as much performance as possible. There are fast serialization formats like SBE that people leverage for this as well.
- joas_coder 2y ago> Given the documentation says that this is supposedly to be between JVMs, how do they handle the serialize/deserialize? Your transfer object needs to implement MemorySerializable. Below two examples from CoralRing's GitHub: https://github.com/coralblocks/CoralRing/blob/main/src/main/java/com/coralblocks/coralring/example/ring/Message.java https://github.com/coralblocks/CoralRing/blob/main/src/main/... https://github.com/coralblocks/CoralRing/blob/main/src/main/java/com/coralblocks/coralring/util/PayloadByteBufferMessage.java https://github.com/coralblocks/CoralRing/blob/main/src/main/... The second one effectively allows you to send anything you want (as bytes) through the ring, making CoralRing message agnostic.
- exabrial 2y agoThis is fascinating. I have no idea what something like this would be used for though... what are the use cases?
- tadfisher 2y agoBasically if you want to schedule workers on a separate JVM, but don't want to pay the latency cost of something like a DB-backed queue or a library with some FFI component.
- capmorganbih 2y agoOur kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts
- sriram_malhar 2y agoCan you explain what your issue is with Kafka? What makes it not reliable enough?
- capmorganbih 2y agoKafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages
- jojohohanon 2y agoThere’s a whole conference in Vegas next week that want to sell you something.
- dikei 2y agoSo basically, you want to build a write-ahead-log before writing data to Kafka, and I think you're underestimating the effort to implement a WAL. If you don't have a person that can manage Kafka, you almost definitely don't have the person to maintain a WAL.
- lizhenqi 2y agoJust a suggestion, you can learn about automq (a message queue that has rewritten the storage layer based on Kafka), which has done a lot in automatic recovery and self-balancing, maybe it will help you. If you don't need it, you can ignore this comment.
- kasey_junk 2y agoYou could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.
- kllrnohj 2y agohttps://github.com/coralblocks/CoralRing/blob/1168b047e0183c9195ab48505e43e8fb2faea73d/src/main/java/com/coralblocks/coralring/ring/BlockingRingConsumer.java#L190 https://github.com/coralblocks/CoralRing/blob/1168b047e0183c... Am I missing something here or does the BlockingRingConsumer not actually block? And worse doesn't it just return garbage if poll is called without first checking availableToPoll? The example sure looks like it... https://github.com/coralblocks/CoralRing/blob/main/src/main/java/com/coralblocks/coralring/example/ring/BlockingConsumer.java https://github.com/coralblocks/CoralRing/blob/main/src/main/... Which if so isn't this like 1/4th a library for doing IPC? It doesn't seem to do much itself
- nulld3v 2y agoIt looks like these are more lower level libraries so clients are supposed to block, but they are responsible for doing the blocking themselves. And given these are designed for "ultra-low-latency" systems, I don't think that's a big problem because the best blocking strategy is probably just to spin. It would be nice if the docs were nicer though, considering this is a paid product...
- joas_coder 2y ago> It looks like these are more lower level libraries so clients are supposed to block, but they are responsible for doing the blocking themselves. You busy spin when blocking (fastest) or you can use a WaitStrategy from https://www.github.com/coralblocks/CoralQueue https://www.github.com/coralblocks/CoralQueue. You can see an example here: https://github.com/coralblocks/CoralQueue/blob/main/src/main/java/com/coralblocks/coralqueue/example/waitstrategy/Minimal.java https://github.com/coralblocks/CoralQueue/blob/main/src/main... > And given these are designed for "ultra-low-latency" systems, I don't think that's a big problem because the best blocking strategy is probably just to spin. If you have an isolated and dedicated CPU core for your thread, that's correct. Busy-spinning is the fastest/best strategy. > It would be nice if the docs were nicer though, considering this is a paid product... CoralRing and CoralQueue are open-source and free at GitHub. There are also a lot of documentation and explanations on the GitHub README.me page (front page). The code has also a lot of comments.
- peterldowns 2y agoDoes anyone have suggestions for something like this, but in Golang?
- nwellinghoff 2y agoI don’t get it. How is this advantageous as it’s limited to one machine? Why wouldn’t you just have one jvm running multiple threads? What is the point of having multiple jvm processes interacting through this ring? Can someone enlighten me?
- gunnarmorling 2y agoA few potential reasons for this design coming to mind: - Resource allocation; you might want to give just specific amount of memory, CPU, network I/O to specific modules of a system, which is not really feasible within a single JVM - Resource isolation; e.g. a memory leak in one module of the system will affect just that specific JVM instance but not others (similar to why browsers run tabs in multiple processes); - Upgrades; you can put a new version of one module of the system into place without impacting the others; while the JVM does support this via dynamic classloading (as e.g. used in OSGi or Layrry, https://github.com/moditect/layrry https://github.com/moditect/layrry), this becomes complex quickly, you can create classloader leaks, etc. - Security; You might have (3rd-party) modules you want to keep isolated from the memory, data, config, etc. of other modules; in particular with the removal of the security manager, OS-enforced process isolation is the way to
- dikei 2y agoYeah, I think the HFT guys use CPU pinning a lot: 1 process - 1 CPU, so you'd need multiple processes to take advantage of multicores server.
- gpderetta 2y agoUsually it is 1 thread - 1 CPU. There might be other reasons (address space separation has its own advantages - and disadvantages) to have distinct processes of course.
- kasey_junk 2y agoAlso software design. You can split jvm into those that have to follow strict parameters (eg no allocations) and those that follow more traditional Java patterns.
- pacoverdi 2y agoRelated: https://github.com/pcdv/jocket https://github.com/pcdv/jocket Drop-in replacement for java.net.Socket using shared memory (and optionally, futex for notification)
- sylware 2y agoultra-low-latency and java in the same sentence... masochists?
- joas_coder 2y agoLOL you have a point. But this is subjective. Someone could argue that a highly complex C++ system is masochism. The reality is: a lot of successful market makers, prop trading firms, banks, etc. use Java for ultra-low-latency financial systems. And of course some successful companies use C++ as well.
- kookamamie 2y agoNumbers aren't subjective. High-performance is a measurable claim and shouldn't be entangled with language-complexity matters.
- joas_coder 2y agoTotally agree! My understanding because of the use of the word "masochist" was that it was possible but difficult. Not that one language was better/worse than the other, in terms of performance.
- speed_spread 2y agoNot at all. The constraints for ultra low latency are the same whether you use C++ or Java: No blocking IO, no dynamic memory allocation. Using the right libraries such as this one, Java can do a surprisingly decent job while keeping the code more accessible to a layman developer than if it was C++. You still need C++ for applications where maximum mechanical sympathy is required (cache alignment guarantees, etc.) but if you have low latency / medium volume Java often does the trick.
- joas_coder 2y agoWell said and I couldn't agree more. There are top market makers and banks using Java for a fact. And other C++ firms as well. Some of them are considering or have considered the move to Java. Some have already done this move. Some will never do. I'm certain that it is trivial for a C++ programmer to code in Java. The opposite of course is not true. The whole point of Java as a language is to be higher-level than C++. I don't know if people have realized, but with GraalVM it is now possible to compile Java code entirely to native code ahead-of-time, like it is done with C++. Even before GraalVM there was already the -Xcomp option to force JIT compilation in the very first pass. However that does not necessarily mean that AOT is always preferable over JIT. It is not. Runtime profiling information so you can determine the critical path (hot spots) is amazing for some optimizations, such as aggressive inlining.
- deleted 2y ago[deleted]
- 1oooqooq 2y agoit's sad how comfy the world for spawning request is versus when you are on the receiving side.
- BiraIgnacio 2y agoSuper cool, nice work!