The only thing that burned my team on Consul was DNS routing. At high request load, it would not load balance fairly leading to hotspots that could lead to a no

The only thing that burned my team on Consul was DNS routing. At high request load, it would not load balance fairly leading to hotspots that could lead to a node falling over and lead to a thundering herd of node failures. This was several years ago, so maybe they fixed it. I believe it was something to do with a time component to the routing, likely some ill-though-out optimization that failed at scale (one node would be hot for some number of seconds, then some other node would take its place). We got around it by making a routing library that used the (non-exposed) Watcher API to build the membership info for the cluster in memory for each node (instead of relying on DNS), and then did round-robin on the cluster for routing each subsequent request.