5 ms·

Somehow I think the overhead of TCP will be a lot more significant than the overhead of a copy from cache to memory. Wouldn't such a system make more sense for

by arjvik 3y ago

Somehow I think the overhead of TCP will be a lot more significant than the overhead of a copy from cache to memory. Wouldn't such a system make more sense for sending raw memory directly over the wire in some form of custom protocol? Even UDP would probably be able to approximate that.

lionkor 3y ago
TCP lends itself well, I assume, because it: - is supported everywhere - allows massive reads and writes (whereas UDP often has quite small limits before packets are dropped) - is reliable (you wont lose data, your data won't corrupt, etc) So you could use this inside an existing datacenter, with existing hardware, and existing software to debug and monitor it. If you send GB of memory, the TCP overhead is minimal.
- klabb3 3y ago
  Tcp is also extremely battletested and performs well not just theoretically but in the face of other traffic (congestion) and in many different environments. Beating TCP reliability and performance is an extremely ambitious endeavor in practice. Add wide interop to that list and it becomes next to impossible.
  - rewmie 3y ago
    It's also important to have an idea what are the theoretical gains to be had with this work. It would make no sense to mess with infrastructure at this level of the tech stack if the ballpark estimates of potential gains are in the single-digit percent points. In this scenario it would be much more efficient to either go with horizontal scaling or look into the possibility of shaving off some requests.
    - jeffbee 3y ago
      Sorry, no. At this scale, if google thought they could shave single digits off their cpu costs they would staff a team to do it. If they thought the digit was “9” they would put dozens of people on it.
      - rewmie 3y ago
        > At this scale, if google (...) How many companies do you know who operate nearly as many servers as Google?
toast0 3y ago
Have a look at Netflix's 800Gbps CDN node presentation. If you want to hit high transfer speeds, your CPU may not have the bandwidth to read from your device and write it to the NIC. DMA between devices (assuming that's what is happening here), saves memory bandwidth for other stuff.
- rewmie 3y ago
  > Have a look at Netflix's 800Gbps CDN node presentation. If you want to hit high transfer speeds, your CPU may not have the bandwidth to read from your device and write it to the NIC. Would it be feasible to horizontal scale those nodes instead?
  - toast0 3y ago
    Sometimes yes, but CDN nodes are competing for space in ISP edge hosting. Fitting an 800Gbps appliance into 2-4U means you can fit into a lot more ISPs. Additionally, if you use multiple hosts in a single deployment, you often also need a router, since the ISPs want to provide a single connection. That adds a significant cost to deployment in dollars and space. Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally. If a bit of engineering time can significantly reduce the number of systems to manage and the hardware budget, seems like a good plan. Especially when these projects are fun to work on and large tech companies want to retain employees.
    - rewmie 3y ago
      > Fitting an 800Gbps appliance into 2-4U means you can fit into a lot more ISPs. The work and investment to turn that 4U deployment into two 2U deployments sounds like a far more realistic and effective way to go about the problem. > Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally. The point is that the capacity does not exist, hence these patches being shot down by kernel maintainers for being fundamentally broken. Also, what are the real-world performance improvements of these memory hacks? Adding another COTS box next to the one already deployed should double capacity straight away.
londons_explore 3y ago
I assume the actual TCP packetization is done by the network hardware in the typical case. Then the overhead of TCP is hidden as long as the network hardware can operate at full line rate.
benlwalker 3y ago
At 200+ Gbps, the copy from LLC where the packet landed to the userspace buffer dominates the performance profiles on most systems. The TCP processing isn't bad and the expensive parts can often be offloaded. I'd contend that this data copy on the TCP recv path is one of the most important performance issues to solve for the entire industry right now. DDIO let us kick the can way down the road, but it seems like the network speeds have outpaced the CPU speeds so much that it can't save us for much longer.
eep_social 3y ago
I think performance limitations are addressed directly by another comment but also consider that UDP might not be available.

by arjvik 3y ago

lionkor 3y ago

TCP lends itself well, I assume, because it: - is supported everywhere - allows massive reads and writes (whereas UDP often has quite small limits before packets are dropped) - is reliable (you wont lose data, your data won't corrupt, etc) So you could use this inside an existing datacenter, with existing hardware, and existing software to debug and monitor it. If you send GB of memory, the TCP overhead is minimal.

klabb3 3y ago
Tcp is also extremely battletested and performs well not just theoretically but in the face of other traffic (congestion) and in many different environments. Beating TCP reliability and performance is an extremely ambitious endeavor in practice. Add wide interop to that list and it becomes next to impossible.
- rewmie 3y ago
  It's also important to have an idea what are the theoretical gains to be had with this work. It would make no sense to mess with infrastructure at this level of the tech stack if the ballpark estimates of potential gains are in the single-digit percent points. In this scenario it would be much more efficient to either go with horizontal scaling or look into the possibility of shaving off some requests.
  - jeffbee 3y ago
    Sorry, no. At this scale, if google thought they could shave single digits off their cpu costs they would staff a team to do it. If they thought the digit was “9” they would put dozens of people on it.
    - rewmie 3y ago
      > At this scale, if google (...) How many companies do you know who operate nearly as many servers as Google?

toast0 3y ago

Have a look at Netflix's 800Gbps CDN node presentation. If you want to hit high transfer speeds, your CPU may not have the bandwidth to read from your device and write it to the NIC. DMA between devices (assuming that's what is happening here), saves memory bandwidth for other stuff.

rewmie 3y ago
> Have a look at Netflix's 800Gbps CDN node presentation. If you want to hit high transfer speeds, your CPU may not have the bandwidth to read from your device and write it to the NIC. Would it be feasible to horizontal scale those nodes instead?
- toast0 3y ago
  Sometimes yes, but CDN nodes are competing for space in ISP edge hosting. Fitting an 800Gbps appliance into 2-4U means you can fit into a lot more ISPs. Additionally, if you use multiple hosts in a single deployment, you often also need a router, since the ISPs want to provide a single connection. That adds a significant cost to deployment in dollars and space. Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally. If a bit of engineering time can significantly reduce the number of systems to manage and the hardware budget, seems like a good plan. Especially when these projects are fun to work on and large tech companies want to retain employees.
  - rewmie 3y ago
    > Fitting an 800Gbps appliance into 2-4U means you can fit into a lot more ISPs. The work and investment to turn that 4U deployment into two 2U deployments sounds like a far more realistic and effective way to go about the problem. > Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally. The point is that the capacity does not exist, hence these patches being shot down by kernel maintainers for being fundamentally broken. Also, what are the real-world performance improvements of these memory hacks? Adding another COTS box next to the one already deployed should double capacity straight away.

londons_explore 3y ago

I assume the actual TCP packetization is done by the network hardware in the typical case. Then the overhead of TCP is hidden as long as the network hardware can operate at full line rate.

benlwalker 3y ago

At 200+ Gbps, the copy from LLC where the packet landed to the userspace buffer dominates the performance profiles on most systems. The TCP processing isn't bad and the expensive parts can often be offloaded. I'd contend that this data copy on the TCP recv path is one of the most important performance issues to solve for the entire industry right now. DDIO let us kick the can way down the road, but it seems like the network speeds have outpaced the CPU speeds so much that it can't save us for much longer.

eep_social 3y ago

I think performance limitations are addressed directly by another comment but also consider that UDP might not be available.