5 ms·

I'm missing one thing from the article, that is commonly missing from performance-related articles. When you talk about playing whack-a-mole with the optimizat

I'm missing one thing from the article, that is commonly missing from performance-related articles. When you talk about playing whack-a-mole with the optimizations, this is what you are missing: > What's the best the hardware can do? You don't say in the article. The article only says that you start at 250k req/s, and ends at 1.2 req/s. Is that good? Is your optimization work done? Can you open a beer and celebrate? The article doesn't say. If the best the hardware can technically do is 1.3M req/s, then you probably can call it a day. But if the best the hardware can do is technically 100M req/s, then you just went from very very bad (0.25% of hardware peak) to just very bad (1.2% of hardware peak). Knowing how many reqs per second should the hardware be able to do is the only way to put things in perspective here.

slver 5y ago
TCP is not typically a hardware feature so how’d you know exactly? Maybe you wanna write a dedicated OS for it? Interesting project but I can’t blame them for not doing it.
- stingraycharles 5y ago
  Offloading TCP to hardware is, in fact, something that is very common, especially once you get into the 10gbit connections area. I would be surprised if AWS didn’t do this.
  - slver 5y ago
    It's available, is it very common, I can't claim. Googling stuff like "Amazon AWS hardware TCP TOE" doesn't reveal anything. So we can't assume that either.
    - jiggawatts 5y ago
      Typically with public cloud vendors you get SR-IOV networking above a certain VM size, but you may have to jump through hoops to enable it. I'm not sure about AWS, but in Azure it is called "Accelerated Networking" and it is available in most recent VM sizes that have 4 CPUs or more. It enables direct hardware connectivity and all offload options. In my testing it reduces latency dramatically, with typical applications seeing a 5x faster small transactions. Similarly, you can get "wire speed" for single TCP streams without any special coding.
    - jeffe 5y ago
      AWS uses an in house tcp-like protocol optimized for distributed computing. Said protocol is implemented on a hardware nic. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9167399 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=916...
- volta83 5y ago
  Your network links supports certain throughput and latencies depending on the packet sizes. Your vendor should tell you what these are, and provide you with benchmarks to reproduce their claims (OSU reproduces these for, e.g., MPI). The network card also has hardware limits in the BW that it can handle, its latency. It is connected with the CPU via PCI-e usually, which has also latency and bandwidths, etc. All this go to the CPU, which has latencies and BW from the different caches and DRAM, etc. So you should be able to model what's the theoretical maximum of request that the network can handle, and then the network interface, the PCI-e bus, etc. up to DRAM. The amount that they can handle differs, so the bottleneck is going to be the slowest part of the chain. For example, as an extremely simplified example, say you have a 100 GB/s network, connected to a network adapter that can handle 200GB/s, connected with PCI-e 3 to the CPU at 12GB/S, which is connected with DRAM at 200GB/s. If each request has to receive or send 1 GB, then you can at most handle 12 req/s because that's all what your PCI-e bus can support. If you are then delivering 1 reqs/s then either your "model" is wrong, or your app is poorly implemented. If you are then delivering 11 req/s, then either your "model" is wrong, or your app is well implemented. But if you are far away from your model, e.g., at 1 reqs/s, you can still validate your model, e.g., by using two PCI-e bus, which you then expect to be 2x as fast. Maybe your data about your PCI-e bw is incorrect, or you are not understanding something about how the packets get transfer, but the model guides you through the hardware bottlenecks. The blog post lacks a "model", and focus on "what the software does" without ever putting it into the context of "what the hardware can do". That is enough to allow you to compare whether software A is faster than software B, but if you are the fastest, it doesn't tell you how far can you go.
  - slver 5y ago
    Handling request response isn’t just about packet count. I might as well claim it’s all just electric current and short some wires for max throughput /s
    - gpderetta 5y ago
      The computation given by parent allows you to compute upper bounds and order of magnitude estimates. He is correct that you need these values to guide your optimizations.
notacoward 5y ago
"What the hardware can do" is not the only useful definition of "done" and doesn't even mean anything for an arbitrarily chosen workload of sufficient complexity for user-level software to be involved. You can open a beer and celebrate when your performance lets you handle X load for less than Y operational cost. Or Z% less than current cost. Masturbating over "wire speed" when the wire has never been the limit for any prior implementation is kind of pointless. P.S. I've worked in both network and storage realms where getting close to wire speed really was a meaningful goal. Probably for longer than some interlocutors' entire careers. But this is not that case.
talawahtech 5y ago
The answer to that question is not quite as straight-forward as you might think. In many ways, this experiment/post is about figuring out the answer to the question of "what is the best the hardware can do". I originally started running these tests using the c5.xlarge (not c5n.xlarge) instance type, which is capable of a maximum 1M packets per second. That is an artificial limit set by AWS at the network hardware level. Now mind you, it is not an arbitrary limit, I am sure they used several factors to decide what limits make the most sense based on the instance size, customer use cases, and overall network health. If I had to hazard a guess I would say that 99% of AWS customers don't even begin to approach that limit, and those that do are probably doing high speed routing and/or using UDP. Virtually no-one would have been hitting 1M req/s with 4 vCPUs doing synchronous HTTP request/response over TCP. Those that did would have been using a kernel bypass solution like DPDK. So this blog post is actually about trying to find "the limit", which is in quotes because it is qualified with multiple conditions: (1) TCP (2) request/response (3) Standard kernel TCP/IP stack. While working on the post, I actively tried to find a network performance testing tool that would let me determine the upper limit for this TCP request/response use case. I looked at netperf, sockperf and uperf (iPerf doesn't do req/resp). For the TCP request/response case they were *all slower* than wrk+libreactor. So it was up to me to find the limit. When I realized that I might hit the 1M req/s limit I switched to the c5n.xlarge whose hardware limit is 1.8M pps. Again, this is just a limit set by AWS. Future tests using a Graviton2 instance + io_uring + recompiling the kernel using profile-guided optimizations might allow us to push past the 1.8M pps limit. Future instances from AWS may just raise the pps limit again... Either way, it should be fun to find out.

slver 5y ago

TCP is not typically a hardware feature so how’d you know exactly? Maybe you wanna write a dedicated OS for it? Interesting project but I can’t blame them for not doing it.

stingraycharles 5y ago
Offloading TCP to hardware is, in fact, something that is very common, especially once you get into the 10gbit connections area. I would be surprised if AWS didn’t do this.
- slver 5y ago
  It's available, is it very common, I can't claim. Googling stuff like "Amazon AWS hardware TCP TOE" doesn't reveal anything. So we can't assume that either.
  - jiggawatts 5y ago
    Typically with public cloud vendors you get SR-IOV networking above a certain VM size, but you may have to jump through hoops to enable it. I'm not sure about AWS, but in Azure it is called "Accelerated Networking" and it is available in most recent VM sizes that have 4 CPUs or more. It enables direct hardware connectivity and all offload options. In my testing it reduces latency dramatically, with typical applications seeing a 5x faster small transactions. Similarly, you can get "wire speed" for single TCP streams without any special coding.
  - jeffe 5y ago
    AWS uses an in house tcp-like protocol optimized for distributed computing. Said protocol is implemented on a hardware nic. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9167399 https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=916...
volta83 5y ago
Your network links supports certain throughput and latencies depending on the packet sizes. Your vendor should tell you what these are, and provide you with benchmarks to reproduce their claims (OSU reproduces these for, e.g., MPI). The network card also has hardware limits in the BW that it can handle, its latency. It is connected with the CPU via PCI-e usually, which has also latency and bandwidths, etc. All this go to the CPU, which has latencies and BW from the different caches and DRAM, etc. So you should be able to model what's the theoretical maximum of request that the network can handle, and then the network interface, the PCI-e bus, etc. up to DRAM. The amount that they can handle differs, so the bottleneck is going to be the slowest part of the chain. For example, as an extremely simplified example, say you have a 100 GB/s network, connected to a network adapter that can handle 200GB/s, connected with PCI-e 3 to the CPU at 12GB/S, which is connected with DRAM at 200GB/s. If each request has to receive or send 1 GB, then you can at most handle 12 req/s because that's all what your PCI-e bus can support. If you are then delivering 1 reqs/s then either your "model" is wrong, or your app is poorly implemented. If you are then delivering 11 req/s, then either your "model" is wrong, or your app is well implemented. But if you are far away from your model, e.g., at 1 reqs/s, you can still validate your model, e.g., by using two PCI-e bus, which you then expect to be 2x as fast. Maybe your data about your PCI-e bw is incorrect, or you are not understanding something about how the packets get transfer, but the model guides you through the hardware bottlenecks. The blog post lacks a "model", and focus on "what the software does" without ever putting it into the context of "what the hardware can do". That is enough to allow you to compare whether software A is faster than software B, but if you are the fastest, it doesn't tell you how far can you go.
- slver 5y ago
  Handling request response isn’t just about packet count. I might as well claim it’s all just electric current and short some wires for max throughput /s
  - gpderetta 5y ago
    The computation given by parent allows you to compute upper bounds and order of magnitude estimates. He is correct that you need these values to guide your optimizations.

notacoward 5y ago

"What the hardware can do" is not the only useful definition of "done" and doesn't even mean anything for an arbitrarily chosen workload of sufficient complexity for user-level software to be involved. You can open a beer and celebrate when your performance lets you handle X load for less than Y operational cost. Or Z% less than current cost. Masturbating over "wire speed" when the wire has never been the limit for any prior implementation is kind of pointless. P.S. I've worked in both network and storage realms where getting close to wire speed really was a meaningful goal. Probably for longer than some interlocutors' entire careers. But this is not that case.

talawahtech 5y ago

The answer to that question is not quite as straight-forward as you might think. In many ways, this experiment/post is about figuring out the answer to the question of "what is the best the hardware can do". I originally started running these tests using the c5.xlarge (not c5n.xlarge) instance type, which is capable of a maximum 1M packets per second. That is an artificial limit set by AWS at the network hardware level. Now mind you, it is not an arbitrary limit, I am sure they used several factors to decide what limits make the most sense based on the instance size, customer use cases, and overall network health. If I had to hazard a guess I would say that 99% of AWS customers don't even begin to approach that limit, and those that do are probably doing high speed routing and/or using UDP. Virtually no-one would have been hitting 1M req/s with 4 vCPUs doing synchronous HTTP request/response over TCP. Those that did would have been using a kernel bypass solution like DPDK. So this blog post is actually about trying to find "the limit", which is in quotes because it is qualified with multiple conditions: (1) TCP (2) request/response (3) Standard kernel TCP/IP stack. While working on the post, I actively tried to find a network performance testing tool that would let me determine the upper limit for this TCP request/response use case. I looked at netperf, sockperf and uperf (iPerf doesn't do req/resp). For the TCP request/response case they were *all slower* than wrk+libreactor. So it was up to me to find the limit. When I realized that I might hit the 1M req/s limit I switched to the c5n.xlarge whose hardware limit is 1.8M pps. Again, this is just a limit set by AWS. Future tests using a Graviton2 instance + io_uring + recompiling the kernel using profile-guided optimizations might allow us to push past the 1.8M pps limit. Future instances from AWS may just raise the pps limit again... Either way, it should be fun to find out.