5 ms·

Different anecdote, but similar vibe.... In ~2010, I was benchmarking Solarflare (Xilinx/AMD now) cards and their OpenOnload kernel-bypass network stack. The

Different anecdote, but similar vibe.... In ~2010, I was benchmarking Solarflare (Xilinx/AMD now) cards and their OpenOnload kernel-bypass network stack. The results showed that two well-tuned systems could communicate faster (lower latency) than two CPU sockets within the same server that had to wait for the kernel to get involved (standard network stack). It was really illuminating and I started re-architecting based on that result. Backing out some of that history... in ~2008, we started using FPGAs to handle specific network loads (US equity market data). It was exotic and a lot of work, but it significantly benefited that use case, both because of DMA to user-land and its filtering capabilities. At that time our network was all 1 Gigabit. Soon thereafter, exchanges started offering 10G handoffs, so we upgraded our entire infrastructure to 10G cut-through switches (Arista) and 10G NICs (Myricom). This performed much better than the 1G FPGA and dramatically improved our entire infrastructure. We then ported our market data feed handler's to Myricom's user-space network stack, because loads were continually increasing and the trading world was continually more competitive... and again we had a narrow solution (this time in software) to a challenging problem. Then about a year later, Solarflare and it's kernel-compatible OpenOnload arrived and we could then apply the power of kernel bypass to our entire infrastructure. After that, the industry returned to FPGAs again with 10G PHY and tons of space to put whole strategies... although I was never involved with that next generation of trading tech. I personally stayed with OpenOnload for all sorts of workloads, growing to use it with containerization and web stacks (Redis, Nginx). Nowadays you can use OpenOnload with XDP; again a narrow technology grows to fit broad applicability.

oogali 4y ago
I went down a similar path (Solarflare, Myricom, and Chelsio; Arista, BNT, and 10G Twinax) and found we could get lower latency internally by confining two OpenOnload-enabled processes that needed to communicate with each other to the same NUMA domain. We architected our applications around that as well. The firm continued on to the FPGA path though that was after my time there. I do still pick up SolarFlare cards off of eBay for home lab purposes.
- neomantra 4y ago
  Nice! We deploy like that too. I document how to do that with Redis here [1]. [1] https://gist.github.com/neomantra/3c9b89887d19be6fa5708bf4017c0ecd https://gist.github.com/neomantra/3c9b89887d19be6fa5708bf401...
  - oogali 4y ago
    Yup, all the same techniques we used (minus Docker but with the addition of disabling Intel Turbo Boost and hyper threading). A few years ago I met with a cloud computing company who was following similar techniques to reduce the noisy neighbor problem and increase performance consistency. Frankly, it’s good to see that there still are bare metal performance specialists out there doing their thing.
tomcam 4y ago
That was such a cool comment that I actually went to http://www.neomantra.com/ http://www.neomantra.com/ and briefly considered applying for a job there and un-retiring
jxy 4y ago
Does containerization have any impact on performance in your use cases?
- neomantra 4y ago
  I've not exhaustively characterized it, but RedHat has this comparison [1]. It is not enough for the scope/scale of my needs. I do still run bare metal workloads though. That said, I have had operational issues in migrating to Docker, which is another sort of performance impact! I reference some of my core isolation issues in that Redis gist and this GitHub issue [2]. [1] https://www.redhat.com/cms/managed-files/201504-onload_containers_brief_v1.0.pdf https://www.redhat.com/cms/managed-files/201504-onload_conta... [2] https://github.com/moby/moby/issues/31086#issuecomment-303744398 https://github.com/moby/moby/issues/31086#issuecomment-30374...
scott_s 4y ago
I discovered something similar in the early to mid 2010s: processes on different Linux systems communicated over TCP faster than processes on the same Linux system. That is, going over the network was faster than on the same machine. The reason was simple: there is a global lock per system for the localhost pseudo-device. Processes communicating on different systems had actual parallelism, because they each had their own network device. Processes communicating on the same system were essentially serialized, because they were competing with each other for the same pseudo-device. At the time, Linux kernel developers basically said "Yeah, don't do that" when people brought up the performance problem.
- nine_k 4y ago
  I wonder if creating more pseudo-devices for in-host networking would help. Doesn't Docker do that already, for other purposes?
- kcexn 4y ago
  That makes sense, Linux has many highly performant IPC options that don't involve the network device. Just the time it takes to correctly construct an ethernet frame is not negligible.
  - mcculley 4y ago
    Processes communicating over TCP via localhost should not need ethernet frames.
    - neomantra 4y ago
      OpenOnload supports user-space acceleration of pipes. For our custom applications, we initiate connections over unaccelerated UNIX sockets and then "upgrade" them to accelerated pipes. It's like setting up pairs of shared-memory queues, but all operated via POSIX calls including the ability to watch them with epoll. This a real old example of this technique using boost::asio [1]. We adapted that from the one in our own reactor library to Onload-pipe-accelerate some OSS that used ASIO. > Just the time it takes to correctly construct an ethernet frame is not negligible. For that reason, Onload/TCPDirect (and other vendors too) have APIs that allow you to pre-allocate and prepare packets (headers and payload), then you can do some final tweaks -- like choosing a ticker and price in an order packet -- and then send it out with minimal latency. [1] https://github.com/neomantra/asio-pipe-transport https://github.com/neomantra/asio-pipe-transport
throw6554 4y ago
That reminds me of the when the openWRT and other open source guys were complaining that the home gateways of the time did not have a big enough CPU to max out the uplink (10-100mbps at the time), and instead built in hardware accelerators. What they did not know was that the hw accelerator was merely an even smaller CPU running a proprietary network stack.
- wtallis 4y ago
  That doesn't sound quite right to me. The worst CPU bottlenecks I recall for consumer routers running OpenWRT were mostly when doing traffic shaping (to compensate for bufferbloat in ISP equipment), not a complete inability to forward traffic fast enough. And when line speed really wasn't achievable with software routing, the hardware deficiency that mattered most was probably memory bandwidth rather than CPU core performance. The kind of offload most prevalent on consumer routers is NAT performed by the Ethernet switch ASIC, which I don't think can be fairly described as an even smaller CPU running a proprietary network stack—you might have a microcontroller-class CPU core in the switch chip, but there's a separation of control and data planes.
easytiger 4y ago
I've worked quite extensively with openonload. There were significant numbers of "free wins" just from LD_PRELOAD though anything non trivial did require more configuration (mostly via env vars!). I've also written ef_vi applications for udp/mc pub in a similar market space to the one you describe. Whilst a little quirky in places you could write some incredibly fast stuff. on 2.xGHz xeon cpus you could send a useful datagram in 600-700ns. For a lot of applications you could even leave that in your critical path and not care.
d4a 4y ago
Seems like you did something similar to what Vector Packet Processing is. https://en.wikipedia.org/wiki/Vector_Packet_Processing https://en.wikipedia.org/wiki/Vector_Packet_Processing

oogali 4y ago

I went down a similar path (Solarflare, Myricom, and Chelsio; Arista, BNT, and 10G Twinax) and found we could get lower latency internally by confining two OpenOnload-enabled processes that needed to communicate with each other to the same NUMA domain. We architected our applications around that as well. The firm continued on to the FPGA path though that was after my time there. I do still pick up SolarFlare cards off of eBay for home lab purposes.

neomantra 4y ago
Nice! We deploy like that too. I document how to do that with Redis here [1]. [1] https://gist.github.com/neomantra/3c9b89887d19be6fa5708bf4017c0ecd https://gist.github.com/neomantra/3c9b89887d19be6fa5708bf401...
- oogali 4y ago
  Yup, all the same techniques we used (minus Docker but with the addition of disabling Intel Turbo Boost and hyper threading). A few years ago I met with a cloud computing company who was following similar techniques to reduce the noisy neighbor problem and increase performance consistency. Frankly, it’s good to see that there still are bare metal performance specialists out there doing their thing.

tomcam 4y ago

That was such a cool comment that I actually went to http://www.neomantra.com/ http://www.neomantra.com/ and briefly considered applying for a job there and un-retiring

jxy 4y ago

Does containerization have any impact on performance in your use cases?

neomantra 4y ago
I've not exhaustively characterized it, but RedHat has this comparison [1]. It is not enough for the scope/scale of my needs. I do still run bare metal workloads though. That said, I have had operational issues in migrating to Docker, which is another sort of performance impact! I reference some of my core isolation issues in that Redis gist and this GitHub issue [2]. [1] https://www.redhat.com/cms/managed-files/201504-onload_containers_brief_v1.0.pdf https://www.redhat.com/cms/managed-files/201504-onload_conta... [2] https://github.com/moby/moby/issues/31086#issuecomment-303744398 https://github.com/moby/moby/issues/31086#issuecomment-30374...

scott_s 4y ago

I discovered something similar in the early to mid 2010s: processes on different Linux systems communicated over TCP faster than processes on the same Linux system. That is, going over the network was faster than on the same machine. The reason was simple: there is a global lock per system for the localhost pseudo-device. Processes communicating on different systems had actual parallelism, because they each had their own network device. Processes communicating on the same system were essentially serialized, because they were competing with each other for the same pseudo-device. At the time, Linux kernel developers basically said "Yeah, don't do that" when people brought up the performance problem.

nine_k 4y ago
I wonder if creating more pseudo-devices for in-host networking would help. Doesn't Docker do that already, for other purposes?
kcexn 4y ago
That makes sense, Linux has many highly performant IPC options that don't involve the network device. Just the time it takes to correctly construct an ethernet frame is not negligible.
- mcculley 4y ago
  Processes communicating over TCP via localhost should not need ethernet frames.
  - neomantra 4y ago
    OpenOnload supports user-space acceleration of pipes. For our custom applications, we initiate connections over unaccelerated UNIX sockets and then "upgrade" them to accelerated pipes. It's like setting up pairs of shared-memory queues, but all operated via POSIX calls including the ability to watch them with epoll. This a real old example of this technique using boost::asio [1]. We adapted that from the one in our own reactor library to Onload-pipe-accelerate some OSS that used ASIO. > Just the time it takes to correctly construct an ethernet frame is not negligible. For that reason, Onload/TCPDirect (and other vendors too) have APIs that allow you to pre-allocate and prepare packets (headers and payload), then you can do some final tweaks -- like choosing a ticker and price in an order packet -- and then send it out with minimal latency. [1] https://github.com/neomantra/asio-pipe-transport https://github.com/neomantra/asio-pipe-transport

throw6554 4y ago

That reminds me of the when the openWRT and other open source guys were complaining that the home gateways of the time did not have a big enough CPU to max out the uplink (10-100mbps at the time), and instead built in hardware accelerators. What they did not know was that the hw accelerator was merely an even smaller CPU running a proprietary network stack.

wtallis 4y ago
That doesn't sound quite right to me. The worst CPU bottlenecks I recall for consumer routers running OpenWRT were mostly when doing traffic shaping (to compensate for bufferbloat in ISP equipment), not a complete inability to forward traffic fast enough. And when line speed really wasn't achievable with software routing, the hardware deficiency that mattered most was probably memory bandwidth rather than CPU core performance. The kind of offload most prevalent on consumer routers is NAT performed by the Ethernet switch ASIC, which I don't think can be fairly described as an even smaller CPU running a proprietary network stack—you might have a microcontroller-class CPU core in the switch chip, but there's a separation of control and data planes.

easytiger 4y ago

I've worked quite extensively with openonload. There were significant numbers of "free wins" just from LD_PRELOAD though anything non trivial did require more configuration (mostly via env vars!). I've also written ef_vi applications for udp/mc pub in a similar market space to the one you describe. Whilst a little quirky in places you could write some incredibly fast stuff. on 2.xGHz xeon cpus you could send a useful datagram in 600-700ns. For a lot of applications you could even leave that in your critical path and not care.

d4a 4y ago

Seems like you did something similar to what Vector Packet Processing is. https://en.wikipedia.org/wiki/Vector_Packet_Processing https://en.wikipedia.org/wiki/Vector_Packet_Processing