My team spent weeks using log-aggregated metrics to gradually figure out why servers' clocks would go out of whack. It turned out Docker Swarm made undocumente

My team spent weeks using log-aggregated metrics to gradually figure out why servers' clocks would go out of whack. It turned out Docker Swarm made undocumented† use of a UDP port that some VMware product also used, and once in a while they'd cross the streams. We only figured it out because we put every system event we could find onto a Grafana graph and narrowed down which ones kept happening at the same time. † I think? It's been a while, might have just been hard to find.