4 ms·

This seems a bit surprising: what is the latency coming from the NIC to the motherboard ? I somehow doubt the latency due to distance within one piece of hardwa

by cdavid 16y ago

This seems a bit surprising: what is the latency coming from the NIC to the motherboard ? I somehow doubt the latency due to distance within one piece of hardware matters (current CPU cannot do much more than a few 1000s cycles for the electricity to go around a few times in a motherboard, which is nothing to do any kind of advanced processing).

bnoordhuis 16y ago
The problem is that a high-bandwidth link overruns the CPU with thousands of interrupts per second. It can paralyse the system, where the CPU is so busy handling I/O that it doesn't get around to actually scheduling processes. Disclaimer: I sometimes consult for HFTs but we don't use FPGAs (yet).
- cdavid 16y ago
  So it is more an architecture issue with conventional hardware ? It makes much more sense to me with this explanation - the issue is the same as in most real-time systems then (like in embedded hardware for audio gears, which is more familiar to me than HFT). Don't things like real time linux or soft real time patches for linux help, though ?
  - bnoordhuis 16y ago
    > Don't things like real time linux or soft real time patches for linux help, though ? I don't see how it could. The CPU essentially drops everything it's doing when a packet comes in. Have that happen often enough and there won't be time left to do anything else, unless you start dropping packets.
    - MichaelGG 16y ago
      Doesn't interrupt moderation and receive-side scaling (new Intel NICs can even do rule-based queue distribution) sort this out?
    - nupark2 16y ago
      We see this behavior on high pps routers -- but isn't this why you have multiple CPUs and thread-based interrupt handling?
larsberg 16y ago
It's pretty terrible (in HFT terms), particularly if you're trying to use a junky on-board NIC. Folks not using FPGAs use separate ethernet cards and enable kernel bypass and can do significantly better with that.

by cdavid 16y ago

bnoordhuis 16y ago

The problem is that a high-bandwidth link overruns the CPU with thousands of interrupts per second. It can paralyse the system, where the CPU is so busy handling I/O that it doesn't get around to actually scheduling processes. Disclaimer: I sometimes consult for HFTs but we don't use FPGAs (yet).

cdavid 16y ago
So it is more an architecture issue with conventional hardware ? It makes much more sense to me with this explanation - the issue is the same as in most real-time systems then (like in embedded hardware for audio gears, which is more familiar to me than HFT). Don't things like real time linux or soft real time patches for linux help, though ?
- bnoordhuis 16y ago
  > Don't things like real time linux or soft real time patches for linux help, though ? I don't see how it could. The CPU essentially drops everything it's doing when a packet comes in. Have that happen often enough and there won't be time left to do anything else, unless you start dropping packets.
  - MichaelGG 16y ago
    Doesn't interrupt moderation and receive-side scaling (new Intel NICs can even do rule-based queue distribution) sort this out?
  - nupark2 16y ago
    We see this behavior on high pps routers -- but isn't this why you have multiple CPUs and thread-based interrupt handling?

larsberg 16y ago

It's pretty terrible (in HFT terms), particularly if you're trying to use a junky on-board NIC. Folks not using FPGAs use separate ethernet cards and enable kernel bypass and can do significantly better with that.