I remember an article--I think from Gitlab--about random latency and out of order packet issues they were seeing on GCP. Turns out it only happened under a cert

I remember an article--I think from Gitlab--about random latency and out of order packet issues they were seeing on GCP. Turns out it only happened under a certain amount of load since GCP's routing algorithm uses a different network if traffic is under a certain amount and switches to a more distributed mode when it crosses a threshold. In addition, sometimes you land on a bad piece of hardware but it's not bad enough to trigger the provider's monitoring. Few months ago we had a bad EC2 instance whose network would drop every 2 hours causing a bunch of random errors before recovering for a little while.