5 ms·

Regarding hardware: I try to not share infrastructure. What I can not fully control can and will dysfunction, even in datacenters. For example, the LAN options

Regarding hardware: I try to not share infrastructure. What I can not fully control can and will dysfunction, even in datacenters. For example, the LAN options are most often VLAN. So for my servers, I have extra NICs and direct connections one-to-one by crossover cable and ARP monitoring. It costs more, but it quickly pays for itself. For software, everything is done in house. As you said, it doesn't have to be done a way or another.

walrus01 8y ago
I was under the impression that the most serious HFT doesn't even use Ethernet anymore. At each end, where it meets a common buy/sell platform, yes. But not for the competing HFT data link companies' inter-city messaging. For the shortwave links in place between NY-Tokyo, for several I'm privy to details of, people have written their own custom dedicated purpose protocols that implement very low bitrate, ultra low latency data messaging with less overhead and ASIC packet processing time.
- devereaux 8y ago
  Depends on what you do. For what I do, Ethernet is acceptable as long as the latency is below 1 ms. Here're some quick stats for what I consider good enough this specific usecase: rtt min/avg/max/std-dev = 0.069/0.095/0.121/0.016 ms Generally it is better to keep things simple and use existing stuff whenever possible. If we are talking about very low latency (say below 30 us) you may need many other things first, like a good timesource. Then indeed, when you start rolling your own stratum1 NTP servers, rolling your own protocols can be a sensible answer to the problem you have.
  - MuffinFlavored 8y ago
    At a high level, what do you do with HFT? You aren't doing the trading, right? You just manage the systems behind it? So what is? Are there really bots "in production" that are making buying and selling decisions tied to real money so quickly that <1ms is needed?
    - walrus01 8y ago
      I'm not directly involved with it, but tangentially at a network engineering level. In things like the NYC-Chicago path, 1 millisecond is an ETERNITY. The actual people working for HFT firms are probably not going to provide much insight into what's going on behind the scenes, due to very restrictive NDAs, security policies and litigious corporate overlords.
    - dmurray 8y ago
      The advantage is being first. If you learn in Tokyo that a stock is going to go up, you can buy all of the available stock in New York, or cancel your order to sell it. It's going to take you 50 ms to get that information to New York, but if your competitor takes 49 ms, you lose money; if he takes 51 ms, you make money. So 1 ms is important, and so is any time difference big enough to outweigh the noise in the exchange's ordering of events. I know people measuring and caring about speed gains of single-digit nanoseconds, roughly equivalent to replacing your two-metre cables with one-metre cables. The end customer (whoever actually wants to buy stocks) doesn't need the price to update every millisecond. But no one has yet figured out a better way to run an electronic exchange than "first come, first served". Source: also worked in HFT for a long time.
  - kevin_thibedeau 8y ago
    Why would you screw around with NTP when you could just use PTP capable infrastructure?
    - devereaux 8y ago
      I have my reasons. I see PTP as a good complement to NTP.
- itchyouch 8y ago
  Microwave and fiber for long haul. Ethernet is still viable in the LAN. With solarflare/mellanox nics + kernel bypass you can get a message from wire the memory in around 2-3 microseconds. The layer 3 10/40g ultra low latency switch offerings can pass a frame port to port in 200ns while layer 2 offering can do it in 5-50ns depending on how many features are stripped away. HFT firms use FPGAs with integrated nics as well. Pci-e Latency to pass a message is around 800ns. For the ULL crowd, when the money opportunity arises, everyone is stampeding to that order to get there first and causes a micro/nanosecond level spike of messaging/activity. FPGAs are great at staying deterministic during this burst. But alas, i digress. Ethernet is still used and can be wicked fast.

walrus01 8y ago

I was under the impression that the most serious HFT doesn't even use Ethernet anymore. At each end, where it meets a common buy/sell platform, yes. But not for the competing HFT data link companies' inter-city messaging. For the shortwave links in place between NY-Tokyo, for several I'm privy to details of, people have written their own custom dedicated purpose protocols that implement very low bitrate, ultra low latency data messaging with less overhead and ASIC packet processing time.

devereaux 8y ago
Depends on what you do. For what I do, Ethernet is acceptable as long as the latency is below 1 ms. Here're some quick stats for what I consider good enough this specific usecase: rtt min/avg/max/std-dev = 0.069/0.095/0.121/0.016 ms Generally it is better to keep things simple and use existing stuff whenever possible. If we are talking about very low latency (say below 30 us) you may need many other things first, like a good timesource. Then indeed, when you start rolling your own stratum1 NTP servers, rolling your own protocols can be a sensible answer to the problem you have.
- MuffinFlavored 8y ago
  At a high level, what do you do with HFT? You aren't doing the trading, right? You just manage the systems behind it? So what is? Are there really bots "in production" that are making buying and selling decisions tied to real money so quickly that <1ms is needed?
  - walrus01 8y ago
    I'm not directly involved with it, but tangentially at a network engineering level. In things like the NYC-Chicago path, 1 millisecond is an ETERNITY. The actual people working for HFT firms are probably not going to provide much insight into what's going on behind the scenes, due to very restrictive NDAs, security policies and litigious corporate overlords.
  - dmurray 8y ago
    The advantage is being first. If you learn in Tokyo that a stock is going to go up, you can buy all of the available stock in New York, or cancel your order to sell it. It's going to take you 50 ms to get that information to New York, but if your competitor takes 49 ms, you lose money; if he takes 51 ms, you make money. So 1 ms is important, and so is any time difference big enough to outweigh the noise in the exchange's ordering of events. I know people measuring and caring about speed gains of single-digit nanoseconds, roughly equivalent to replacing your two-metre cables with one-metre cables. The end customer (whoever actually wants to buy stocks) doesn't need the price to update every millisecond. But no one has yet figured out a better way to run an electronic exchange than "first come, first served". Source: also worked in HFT for a long time.
- kevin_thibedeau 8y ago
  Why would you screw around with NTP when you could just use PTP capable infrastructure?
  - devereaux 8y ago
    I have my reasons. I see PTP as a good complement to NTP.
itchyouch 8y ago
Microwave and fiber for long haul. Ethernet is still viable in the LAN. With solarflare/mellanox nics + kernel bypass you can get a message from wire the memory in around 2-3 microseconds. The layer 3 10/40g ultra low latency switch offerings can pass a frame port to port in 200ns while layer 2 offering can do it in 5-50ns depending on how many features are stripped away. HFT firms use FPGAs with integrated nics as well. Pci-e Latency to pass a message is around 800ns. For the ULL crowd, when the money opportunity arises, everyone is stampeding to that order to get there first and causes a micro/nanosecond level spike of messaging/activity. FPGAs are great at staying deterministic during this burst. But alas, i digress. Ethernet is still used and can be wicked fast.