About 10 years ago, I was working in a startup that had a hadoop cluster on some bare metal servers in a datacenter. At one point when the dataset size and sust

About 10 years ago, I was working in a startup that had a hadoop cluster on some bare metal servers in a datacenter. At one point when the dataset size and sustained traffic between nodes grew to a particular level, the network throughput crashed to a trickle, until the hadoop jobs were all canceled, then network was fine a minute later. It turned out to be pause frames! The switches were too eager to propagate them, or something? The fix was to just disable them on all NICs - TCP can handle packet loss and rate control pretty well in those conditions anyway (gigabit switched ethernet). (I can't take any credit for the work debugging and mitigating this issue.) Anyway, it seems this is one of those bugs that pops back up from time to time... people try to implement this bit of the spec along with the rest, but don't realize how it's really supposed to behave under stress, and it's never tested under this kind of stress...