4 ms·

> Since 1995, we’ve seen an increase in datacenter NIC speeds go from 10Mb to 40Gb, with 50Gb and 100Gb just around the corner. This has been a huge factor fo

> Since 1995, we’ve seen an increase in datacenter NIC speeds go from 10Mb to 40Gb, with 50Gb and 100Gb just around the corner. This has been a huge factor for our firm. Trading systems used to be, by necessity, monolithic in design. Your risk system, feed handlers, portfolio management, strategies, execution management, and drop copy/logging systems were all built into one process due to poor speed of interproceess communication, even on the same vlan. You just couldn't get the required performance and certainty with a distributed system. I mean its nice that your distributed database is eventually consistant over the course of 500 milliseconds, but that's a couple of order of magnitude too slow Now, networks are sooooooo fast and bandwidth is so plentiful that we can afford to split many of the major subsystems onto different machines. It's important to note that we don't split these subsystems up for redundancy, I mean we aren't Netflix, we can't pull out major subsystems and keep trading. if one goes down we intentionally take down the entire system. You can laugh at me here if you'd like.... Rather we split the systems up so that we can test and upgrade them easier. Before the split, each upgrade was a major change that might have happened 2x a month, now we can upgrade individual systems 2-3 times a week. Agian, you may laugh now, but I put a very big premium on stability and certainty. I remember talking to someone at Google who told me that their low latency RPC framework was considered to be one of their key technological advantages. Having made the move over the course of a couple of years, I'd be inclined to agree, a robust and fast RPC framework is an incredibly freeing piece of technology.

footiethrowa 11y ago
> I remember talking to someone at Google who told me that their low latency RPC framework was considered to be one of their key technological advantages. Having made the move over the course of a couple of years, I'd be inclined to agree, a robust and fast RPC framework is an incredibly freeing piece of technology. Have you found Google's RPC framework to be robust and fast, or something else? And if so, would you mind sharing what it is? Thanks!
- _yy 11y ago
  Protobuf, probably.
- chollida1 11y ago
  We've experimented with Protocol Buffers, Thrift and CapnProto. We settled on protobuf but capn proto was super fast. I just didn't have the time to play around with it or fortitude to put it into production with it being so new. I should point out, that's my own hang up and not mean to disparage Capn Proto at all, call it the web 2.0 of "Nobody ever got fired for buying IBM".
  - kanslanf 11y ago
    I really like what I've read of CapnProto and would try it for my next distributed system project.
    - corysama 11y ago
      CapnProto has good competition from Google's Flatbuffers. Which one is better comes down to the specifics of your data. https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... https://news.ycombinator.com/item?id=7901991 https://news.ycombinator.com/item?id=7901991
- rictic 11y ago
  The newest version was actually open sourced earlier this year: http://www.grpc.io/ http://www.grpc.io/
Florin_Andrei 11y ago
> You can laugh at me here if you'd like Why laugh? Different businesses, different requirements, different solutions. Horses for courses, as they say.
MichaelGG 11y ago
If a single scaled up server can actually handle the load, why introduce a network? Obviously cross-network comms can't be faster than cross-process comms. And with current NUMA machines, you have a system that's really more like a cluster of computers anyways (with a fast inter-proc but still slower than intracore). What's the real benefit of using RDMA or something to write to a foreign machine's memory versus nontemporal writes to a foreign socket's memory? BTW any suggested reading on the technical implementation design of trading systems? I read that Island's system was distributed (horizontally scalable) even in the 90s. Is that doing things like running separate trading instances per symbol? (Which should work unless there's atomicity required across symbols, like in spread orders?)
- openasocket 11y ago
  I have no idea about the OP's system, but I can see some advantages to breaking components up across a network, at least for vertical scaling. First is that server prices do not scale linearly: four 10-core machines will generally be cheaper than a single 40-core machine. Second is that some programs can hog memory/CPU time; forcing programs to run on their own machine can ease some of that contention. These gains can seem rather small, but can make a big difference at large scales. If you don't mind the drop in latency it's a perfectly acceptable architecture.
  - noir_lord 11y ago
    Third advantage is it forces proper separation, it can be very tempting to do things you shouldn't to local resources for "speed".

footiethrowa 11y ago

> I remember talking to someone at Google who told me that their low latency RPC framework was considered to be one of their key technological advantages. Having made the move over the course of a couple of years, I'd be inclined to agree, a robust and fast RPC framework is an incredibly freeing piece of technology. Have you found Google's RPC framework to be robust and fast, or something else? And if so, would you mind sharing what it is? Thanks!

_yy 11y ago
Protobuf, probably.
chollida1 11y ago
We've experimented with Protocol Buffers, Thrift and CapnProto. We settled on protobuf but capn proto was super fast. I just didn't have the time to play around with it or fortitude to put it into production with it being so new. I should point out, that's my own hang up and not mean to disparage Capn Proto at all, call it the web 2.0 of "Nobody ever got fired for buying IBM".
- kanslanf 11y ago
  I really like what I've read of CapnProto and would try it for my next distributed system project.
  - corysama 11y ago
    CapnProto has good competition from Google's Flatbuffers. Which one is better comes down to the specifics of your data. https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... https://news.ycombinator.com/item?id=7901991 https://news.ycombinator.com/item?id=7901991
rictic 11y ago
The newest version was actually open sourced earlier this year: http://www.grpc.io/ http://www.grpc.io/

Florin_Andrei 11y ago

> You can laugh at me here if you'd like Why laugh? Different businesses, different requirements, different solutions. Horses for courses, as they say.

MichaelGG 11y ago

If a single scaled up server can actually handle the load, why introduce a network? Obviously cross-network comms can't be faster than cross-process comms. And with current NUMA machines, you have a system that's really more like a cluster of computers anyways (with a fast inter-proc but still slower than intracore). What's the real benefit of using RDMA or something to write to a foreign machine's memory versus nontemporal writes to a foreign socket's memory? BTW any suggested reading on the technical implementation design of trading systems? I read that Island's system was distributed (horizontally scalable) even in the 90s. Is that doing things like running separate trading instances per symbol? (Which should work unless there's atomicity required across symbols, like in spread orders?)

openasocket 11y ago
I have no idea about the OP's system, but I can see some advantages to breaking components up across a network, at least for vertical scaling. First is that server prices do not scale linearly: four 10-core machines will generally be cheaper than a single 40-core machine. Second is that some programs can hog memory/CPU time; forcing programs to run on their own machine can ease some of that contention. These gains can seem rather small, but can make a big difference at large scales. If you don't mind the drop in latency it's a perfectly acceptable architecture.
- noir_lord 11y ago
  Third advantage is it forces proper separation, it can be very tempting to do things you shouldn't to local resources for "speed".