3 ms·
Yes - I work in an investment bank, we try to do millisecond-level latencies for our order management system that sends client orders to the various exchanges(f
by xwolfi 5y ago
Yes - I work in an investment bank, we try to do millisecond-level latencies for our order management system that sends client orders to the various exchanges(for sub-milli we use FPGA but it's very expensive and only for some clients).
It works alright (like you I hated it all before, coming from more amateurish implementations). It's slow to change (adding a new event type can take years before it works everywhere), failures can happen - for instance if the message parsing library has a crushing bug and an extendable attribute zone has a poison pill that never appeared before - well nothing you can do but manually editing the event source.
What it brings us, I suppose, is that every micro service is single threaded, all events are well-recorded , we use multicast to transfer them from one sequencer to all consumers so we just need good routers and TCP-level message building - it's very barebone to keep it fast, extensibility for us is mainly on adding more services around a core stream that we don't really need to change all that much (we do a lot of regulatory validations, data analysis, the odd scale out for a round-robin compatible process - not all of them are, some need to see all events, for instance for cumulative exposure calculation - client stock exposure on several markets for risk-based decisions).
It also avoid latencies like in state-request based systems, since each service will build its own state machine. We make a lot of money on this system, so we can hire hundreds of people around the world to maintain it.
At this point I dont see how to do it better (5ms round trip to the client if low validation, 100ms if crossing seawater to an exchange with short selling validations) without events, but I know very well that doing it for simpler flows that are not latency-constrained will probably result in heavy cost and low gain. I would never recommend it for people who aren't already struggling with a fully-fledged business implementation that makes money they want to accelerate.
The problems we face are:
- it's slow to evolve the very core of the system
- we need perfect ordering, there will always be time wasted at the sequencer to transform unordered "commands" for the various services into perfectly ordered events
- testing and debugging is an art that takes time to acquire: I can now, but the task seemed daunting when I started - how to spin up the minimal surface of services to make a valuable replay, how to make static configuration reproduce production's behavior exactly so that all services behave the way you want to reproduce if you're investigating a sequence-based issue (rather than a function-based issue)
- it takes up to a year for a new Java dev to get productive on such an exotic mindset, but it's also because we do no intraday malloc since we cannot afford a GC in the middle of a client order
- management cannot understand why they can't cut cost using the cloud, virtual machines, vendor databases etc. Even in a company that makes billions over 20 years on this system, we still can't explain it in a way that sounds valuable vs its cost. Because its cost probably is extremely high, and can't be outsourced by hit and run consulting managers before being brought in-house again. So we're not like the most popular dev, we're the slow and expensive ones :(
- mightymosquito 5y agoHey thanks for you answer, this is pretty informative. I am basically where you began i.e don't really know what an actual event driven system actually looks like(as in something implemented properly). 1. Could you please tell me what according to you was a major difference between the more amateurish implementation vs the proper implementation(I know this is quite a general question to ask, sorry about that). 2. Since you've seen what proper looks like when it comes to event driven architecture, can you perhaps suggest some resource(could be a book, could be code base) which according to you is the closest someone can get to understanding how a proper event driven system would be implemented.