5 ms·

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would move to RDMA over infiniband. For shared memory clusters, thats what's been done for years. but for general purpose computing its pretty rare. Most of the time its not worth the effort. Infiniband is cheap now, so its not that hard to deploy RDMA[1] type interconnects. Having a reliable layer2 with inbuilt flow control solves a number of issues, even if you are just slamming IP over the top. shit, even 25/100gig is cheap now. so most of your problems can be solved by putting extra nics in your servers and have fancypants distributed LACP type setups on your top of rack/core network. The biggest issue is that its not the network that's constraining throughput, it either processing or some other non network IO. [1]I mean it is hard, but not as hard as implementing a brand new protocol and expecting it to be usable and debuggable.

amluto 4y ago
The paper explicitly addresses Infiniband.
- KaiserPro 4y ago
  not really. they conflate infiniband with RoCE which given they have different semantics on congestion control, I'd say is a bit of a whoopsey. if they are using RoCE, are they using DCB to avoid loss(well make it "lossless")? the paper implies otherwise.
  - birdyrooster 4y ago
    For those who don’t know, RoCE is somewhat of a failure in the marketplace right now.
    - throw0101a 4y ago
      For those who don't know, RoCE = RDMA over Converged Ethernet. * https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet > RDMA over Converged Ethernet (RoCE) is a network protocol that leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. RoCE incorporates the IBTA RDMA semantics to allow devices to perform direct memory-to-memory transfers at the application level without involving the host CPU. Both the transport processing and the memory translation and placement are performed by the hardware which enables lower latency, higher throughput, and better performance compared to software-based protocols. * https://docs.nvidia.com/networking/pages/viewpage.action?pageId=19807245 https://docs.nvidia.com/networking/pages/viewpage.action?pag...
- bayindirh 4y ago
  IB does not work TCP/IP by default. You can either run TCP over IB, which has a performance penalty, or you can directly run in Ethernet mode, which is something completely different.
pclmulqdq 4y ago
The reality of today's large datacenters is that almost all of them have almost all of their traffic on TCP unless the owners of the datacenter have made a conscious effort to not use TCP. The highest-traffic applications, usually databases and storage systems, pretty much all use TCP unless you are buying a purpose-built HPC scale-out storage system (like a Lustre cluster). Most people who build a datacenter today use databases or object stores for storage, not Lustre or dedicated fiber channel SANs. On top of that, pub/sub systems all use TCP today, logging tends to be TCP, etc.
- KaiserPro 4y ago
  Fibre channel is dead, long live fibre channel. I agree a lot of things are on TCP, but I don't think its a massive problem, unless you are running close to the limit of your core network. And one solution to that is to upgrade your core network.... Failing that, implementing some load balancing/partitioning systems to make sure data-processing affinity is best matched. This the better solution, because it yields other advantages as well. But its not the easiest, unless you have a good scheduler
  - pclmulqdq 4y ago
    I will also add that one of the big problems with TCP is that it is impossible to load balance without knowledge of the L4 protocol. You can't load balance a byte stream. That means writing your own load balancer unless you want to also accept http overhead.
  - still_grokking 4y ago
    What is supposed to be a good scheduler? Genuine question as I (as a software dev) have no clue how modern DCs are build in detail.
    - KaiserPro 4y ago
      Its very much down to your workload and how you want it to work. Short answer: k8s/fargate/ECS/batch will do what most people want. Personally I'd steer clear of k8s until you 100% need that overhead. Managed services are ok. Long answer: K8s has a whole bunch of scheduling algorithms but its a jack of all trades, and only really deals with very shallow dependencies (there are plugins but I've not used them). For < 100 machines it works well enough. (by machine I mean either a big VM or physical machine). Once you get more than that, K8S is a bit of a liability. its chatty, slow and noisy. It also is very opinionated. And let us not start on the networking. Like most things there are tradeoffs. Do you want to prioritise resiliency/uptime over efficiency? do you want to have batch processing? do you want it to manage complex dependencies (ie service x needs 15 other services to run first, which then need 40 other services) are you running on unreliable infra (ie spot instances to save money) do you need to partition services based on security? are you running over many datacentres and need the concept of affinity? More detail: The scheduler/dispatcher is almost always deigned to run the specific types of workload that you as a company run. The caveat being that this only applies if you are running multiple datacentres/regions with thousands of machines. Google have a need to run both realtime and batch processing. But as they have millions of servers, making sure that all machines are running at 100% utilisation (or as close to it as practical) is worth hundreds of millions. Its the same with facebook. Netflix I guess has a slightly different setup as they are network IO orientated, so they are all about data affinity, and cache modelling. For them making sure that the edge serves as much as possible is a real cost saving, as bulk bandwidth transfer is expensive. The rest is all machine learning and transcoding I suspect (but that's a guess)
wmf 4y ago
You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".
- ksec 4y ago
  AWS is True HyperScale. Even more so than Google. And yet their spend more money solution on hardware seems to work fine.
  - fragmede 4y ago
    Do we know for a fact that AWS does or doesn't use TCP on their backend? https://news.ycombinator.com/item?id=33402364 https://news.ycombinator.com/item?id=33402364 leads me to believe Google doesn't.
- KaiserPro 4y ago
  You're missing the point that maybe, just maybe, I'm part of a team that looks after >5 million servers. You might also divine that while TCP can be a problem, a bigger problem is data affinity. Shuttling data from a next door rack costs less than one that's in the next door hall, and significantly less than the datacentre over. With each internal hop, the risk of congestion increases. You might also divine that changing everything from TCP to a new, untested protocol across all services, with all that associated engineering effort, plus translation latency, might not be worth it. Especially as now all your observability and protocol routing tools don't work. quick maths: a faster top of rack switch is possibly the same cost as 5 days engineering wage for a mid level google employee. How many new switches do you think you could buy with the engineering effort required to port everything to the new protocol, and have it stable and observable? As a side note "oh but they are google" is not a selling point. Google has google problems half of which are things related to their performance/promotion system which penalises incremental changes in favour of $NEW_THING. HTTP2.0 was also a largely google effort designed to tackle latency over lossy network connections. which it fundamentally didn't do because a whole bunch of people didn't understand how TCP worked and were shocked to find out that mobile performance was shit.
TheRealDunkirk 4y ago
God, I love it when the talk turns hyper-technical around here, and the Jedi masters turn up.
- primeblue 4y ago
counttheforks 4y ago
> Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) So storage doesn't use TCP, except for the protocols that are actually used, which do use TCP?
- KaiserPro 4y ago
  Depends on what you are using, for connecting block stores, you'll use some sort of fabric. That is Fibre channel, SAS, NVME over something or other If you are using GPFS, then you can do stuff over IB, but I don't know how that works. Lustre I imagine does lustre things over RDMA. For everything else, NFS all the things. pNFS means that you can just throw servers at the problem and let the network figure it out. But again, if IO speed is critical, you move IO over to a dedicated fabric of somesort. for most thing NFS is good enough. (except databases, its possible but not great. but then depending on your docker setup, you might be kneecaping your performance because overlayfs is causing io amplification)
colinmhayes 4y ago
> The biggest issue is that its not the network that's constraining throughput, it either processing To be fair the paper talks a bit about how TCP makes multithreading slower compared to a message based system.

amluto 4y ago

The paper explicitly addresses Infiniband.

KaiserPro 4y ago
not really. they conflate infiniband with RoCE which given they have different semantics on congestion control, I'd say is a bit of a whoopsey. if they are using RoCE, are they using DCB to avoid loss(well make it "lossless")? the paper implies otherwise.
- birdyrooster 4y ago
  For those who don’t know, RoCE is somewhat of a failure in the marketplace right now.
  - throw0101a 4y ago
    For those who don't know, RoCE = RDMA over Converged Ethernet. * https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet > RDMA over Converged Ethernet (RoCE) is a network protocol that leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. RoCE incorporates the IBTA RDMA semantics to allow devices to perform direct memory-to-memory transfers at the application level without involving the host CPU. Both the transport processing and the memory translation and placement are performed by the hardware which enables lower latency, higher throughput, and better performance compared to software-based protocols. * https://docs.nvidia.com/networking/pages/viewpage.action?pageId=19807245 https://docs.nvidia.com/networking/pages/viewpage.action?pag...
bayindirh 4y ago
IB does not work TCP/IP by default. You can either run TCP over IB, which has a performance penalty, or you can directly run in Ethernet mode, which is something completely different.

pclmulqdq 4y ago

The reality of today's large datacenters is that almost all of them have almost all of their traffic on TCP unless the owners of the datacenter have made a conscious effort to not use TCP. The highest-traffic applications, usually databases and storage systems, pretty much all use TCP unless you are buying a purpose-built HPC scale-out storage system (like a Lustre cluster). Most people who build a datacenter today use databases or object stores for storage, not Lustre or dedicated fiber channel SANs. On top of that, pub/sub systems all use TCP today, logging tends to be TCP, etc.

KaiserPro 4y ago
Fibre channel is dead, long live fibre channel. I agree a lot of things are on TCP, but I don't think its a massive problem, unless you are running close to the limit of your core network. And one solution to that is to upgrade your core network.... Failing that, implementing some load balancing/partitioning systems to make sure data-processing affinity is best matched. This the better solution, because it yields other advantages as well. But its not the easiest, unless you have a good scheduler
- pclmulqdq 4y ago
  I will also add that one of the big problems with TCP is that it is impossible to load balance without knowledge of the L4 protocol. You can't load balance a byte stream. That means writing your own load balancer unless you want to also accept http overhead.
- still_grokking 4y ago
  What is supposed to be a good scheduler? Genuine question as I (as a software dev) have no clue how modern DCs are build in detail.
  - KaiserPro 4y ago
    Its very much down to your workload and how you want it to work. Short answer: k8s/fargate/ECS/batch will do what most people want. Personally I'd steer clear of k8s until you 100% need that overhead. Managed services are ok. Long answer: K8s has a whole bunch of scheduling algorithms but its a jack of all trades, and only really deals with very shallow dependencies (there are plugins but I've not used them). For < 100 machines it works well enough. (by machine I mean either a big VM or physical machine). Once you get more than that, K8S is a bit of a liability. its chatty, slow and noisy. It also is very opinionated. And let us not start on the networking. Like most things there are tradeoffs. Do you want to prioritise resiliency/uptime over efficiency? do you want to have batch processing? do you want it to manage complex dependencies (ie service x needs 15 other services to run first, which then need 40 other services) are you running on unreliable infra (ie spot instances to save money) do you need to partition services based on security? are you running over many datacentres and need the concept of affinity? More detail: The scheduler/dispatcher is almost always deigned to run the specific types of workload that you as a company run. The caveat being that this only applies if you are running multiple datacentres/regions with thousands of machines. Google have a need to run both realtime and batch processing. But as they have millions of servers, making sure that all machines are running at 100% utilisation (or as close to it as practical) is worth hundreds of millions. Its the same with facebook. Netflix I guess has a slightly different setup as they are network IO orientated, so they are all about data affinity, and cache modelling. For them making sure that the edge serves as much as possible is a real cost saving, as bulk bandwidth transfer is expensive. The rest is all machine learning and transcoding I suspect (but that's a guess)

wmf 4y ago

You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".

ksec 4y ago
AWS is True HyperScale. Even more so than Google. And yet their spend more money solution on hardware seems to work fine.
- fragmede 4y ago
  Do we know for a fact that AWS does or doesn't use TCP on their backend? https://news.ycombinator.com/item?id=33402364 https://news.ycombinator.com/item?id=33402364 leads me to believe Google doesn't.
KaiserPro 4y ago
You're missing the point that maybe, just maybe, I'm part of a team that looks after >5 million servers. You might also divine that while TCP can be a problem, a bigger problem is data affinity. Shuttling data from a next door rack costs less than one that's in the next door hall, and significantly less than the datacentre over. With each internal hop, the risk of congestion increases. You might also divine that changing everything from TCP to a new, untested protocol across all services, with all that associated engineering effort, plus translation latency, might not be worth it. Especially as now all your observability and protocol routing tools don't work. quick maths: a faster top of rack switch is possibly the same cost as 5 days engineering wage for a mid level google employee. How many new switches do you think you could buy with the engineering effort required to port everything to the new protocol, and have it stable and observable? As a side note "oh but they are google" is not a selling point. Google has google problems half of which are things related to their performance/promotion system which penalises incremental changes in favour of $NEW_THING. HTTP2.0 was also a largely google effort designed to tackle latency over lossy network connections. which it fundamentally didn't do because a whole bunch of people didn't understand how TCP worked and were shocked to find out that mobile performance was shit.

TheRealDunkirk 4y ago

God, I love it when the talk turns hyper-technical around here, and the Jedi masters turn up.

primeblue 4y ago

counttheforks 4y ago

> Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) So storage doesn't use TCP, except for the protocols that are actually used, which do use TCP?

KaiserPro 4y ago
Depends on what you are using, for connecting block stores, you'll use some sort of fabric. That is Fibre channel, SAS, NVME over something or other If you are using GPFS, then you can do stuff over IB, but I don't know how that works. Lustre I imagine does lustre things over RDMA. For everything else, NFS all the things. pNFS means that you can just throw servers at the problem and let the network figure it out. But again, if IO speed is critical, you move IO over to a dedicated fabric of somesort. for most thing NFS is good enough. (except databases, its possible but not great. but then depending on your docker setup, you might be kneecaping your performance because overlayfs is causing io amplification)

colinmhayes 4y ago

> The biggest issue is that its not the network that's constraining throughput, it either processing To be fair the paper talks a bit about how TCP makes multithreading slower compared to a message based system.