5 ms·

That makes sense, Linux has many highly performant IPC options that don't involve the network device. Just the time it takes to correctly construct an ethernet

by kcexn 4y ago

That makes sense, Linux has many highly performant IPC options that don't involve the network device. Just the time it takes to correctly construct an ethernet frame is not negligible.

mcculley 4y ago
Processes communicating over TCP via localhost should not need ethernet frames.
- neomantra 4y ago
  OpenOnload supports user-space acceleration of pipes. For our custom applications, we initiate connections over unaccelerated UNIX sockets and then "upgrade" them to accelerated pipes. It's like setting up pairs of shared-memory queues, but all operated via POSIX calls including the ability to watch them with epoll. This a real old example of this technique using boost::asio [1]. We adapted that from the one in our own reactor library to Onload-pipe-accelerate some OSS that used ASIO. > Just the time it takes to correctly construct an ethernet frame is not negligible. For that reason, Onload/TCPDirect (and other vendors too) have APIs that allow you to pre-allocate and prepare packets (headers and payload), then you can do some final tweaks -- like choosing a ticker and price in an order packet -- and then send it out with minimal latency. [1] https://github.com/neomantra/asio-pipe-transport https://github.com/neomantra/asio-pipe-transport
- jchw 4y ago
  In fact, if I recall correctly, it doesn't seem to. I remember writing a tool that analyzed pcap dumps and initially assuming there would be Ethernet frames at the bottom, only to run into the fact that this wasn't true on lo, which had it's own frames, which I assume could vary by OS a bit.
- account42 4y ago
  Processes communicating via localhost should also not need TCP.
  - mcculley 4y ago
    There are many applications which are network transparent and would need extra code to know how to optimize when the other process happens to be on the same machine (e.g., using Sys V IPC or Unix domain sockets). There are good reasons to optimize the case of localhost. I am always bemused when people imagine that communication over localhost involves packets hitting an ethernet adapter.

by kcexn 4y ago

That makes sense, Linux has many highly performant IPC options that don't involve the network device. Just the time it takes to correctly construct an ethernet frame is not negligible.

mcculley 4y ago

Processes communicating over TCP via localhost should not need ethernet frames.

neomantra 4y ago
OpenOnload supports user-space acceleration of pipes. For our custom applications, we initiate connections over unaccelerated UNIX sockets and then "upgrade" them to accelerated pipes. It's like setting up pairs of shared-memory queues, but all operated via POSIX calls including the ability to watch them with epoll. This a real old example of this technique using boost::asio [1]. We adapted that from the one in our own reactor library to Onload-pipe-accelerate some OSS that used ASIO. > Just the time it takes to correctly construct an ethernet frame is not negligible. For that reason, Onload/TCPDirect (and other vendors too) have APIs that allow you to pre-allocate and prepare packets (headers and payload), then you can do some final tweaks -- like choosing a ticker and price in an order packet -- and then send it out with minimal latency. [1] https://github.com/neomantra/asio-pipe-transport https://github.com/neomantra/asio-pipe-transport
jchw 4y ago
In fact, if I recall correctly, it doesn't seem to. I remember writing a tool that analyzed pcap dumps and initially assuming there would be Ethernet frames at the bottom, only to run into the fact that this wasn't true on lo, which had it's own frames, which I assume could vary by OS a bit.
account42 4y ago
Processes communicating via localhost should also not need TCP.
- mcculley 4y ago
  There are many applications which are network transparent and would need extra code to know how to optimize when the other process happens to be on the same machine (e.g., using Sys V IPC or Unix domain sockets). There are good reasons to optimize the case of localhost. I am always bemused when people imagine that communication over localhost involves packets hitting an ethernet adapter.