9 ms·
Nasdaq does go through great pains to ensure that public & private feeds go out "equally"-ish on the wire whether it's a quote going out of the prop or public d
by itchyouch 11y ago
Nasdaq does go through great pains to ensure that public & private feeds go out "equally"-ish on the wire whether it's a quote going out of the prop or public data feeds. At the packet level, a public quote to the SIP or a private quote via Itch are within the sub ~10u level. But once the packet leaves, the fate of that packet is surrendered to the speed of the SIP system.
There's a couple of inherent challenges.
The SIP that aggregates the data is by design a separate system from the trading systems. This is for obvious reasons, as every participant provides their quote to the SIP. So no matter how fast the SIP the is, even with a 99.999% latency of 20 microseconds, the SIP will always be 20 micros + travel time to the SIP (1 to 500 to 7000 micros) as opposed to a direct feed. Nasdaq is in Carteret. NYSE is in Mahwah. BATS is at NY4 in Secaucus, etc. So direct feeds will always be the fastest, even with a SIP that can defy the speed of light a little bit.
Nasdaq does actually read marketdata directly from other exchange participants and falls back to using the "slower SIP" data as a fall back in the event of any issues. This allows for the latest quotes to be respected.
The challenge with the SIPs being hundreds of millis behind is partly due to the criticality & member-driven consensus system in place. Each of the marketcenters NYSE, Nasdaq, Bats, etc have a stake and voting power on how the SIP would be run & operated. Since it's a system that "works" there's really no motivation to continually improve the technology/latency of the SIP. There's also a fear to change the SIP and lead to it going down. If a SIP goes down, by rule, a regulatory halt must be issued and all trading in the securities that particular SIP handles must stop. This is what occurred during the 3 hour outage several years back at Nasdaq. The trading system was fine, but since the SIP was down, everything had to be halted. Halts are detrimental to business since the respective marketcenters make money on trades. Halts = No Trades = Lost Revenue, so there is a lot of motivation to not halt by all the member exchanges.
That ambivalence to a tech refresh has since changed since the scathing critique of Flash Boys and the Nasdaq SIP is in process of being refreshed to the same architecture as the trading systems, which should bring median latencies of the SIP down into the tens of microseconds and 99% latencies into the hundreds of microseconds. Current medians and 90th percentiles are at 450/810 microseconds, which is considered an eternity.
http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-won-over-sip-committee-with-latency-reductions-and-tech-upgrades/d/d-id/1317316 http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-w...
http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Processor%20Stats.pdf http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Pr...
On the technical side, one of the challenges with synchronizing packet delivery between a private & public feed is that the absolute fastest NICs available today with kernel bypass can get a packet from wire to memory or memory to wire in about 2-3u. So every hop of a computer costs a minimum of 4-6u. Usually this is Order Port Ingress -> Matching Engine -> Order Port Egress. An FPGA NIC that does the IP Header processing and delivers data to memory can do wire to memory in about 1-2u. One-way PCI-E latency is around 800 nanos. So there's about a 500-1000 nano budget to process a packet in the FPGA, then another 800 nanos to deliver it over PCI-E to memory to be processed. The fastest cut-through switches have a port-to-port latency of around 250 nanos, so switch hops are pretty negligible in the overall latency of a system, but they do play a part. Anyway, making sure that everything is super duper optimized and running as it should be in a semi-continuously deployed environment makes for interesting and quite crazy/bull-shit level challenges.