3 ms·
"To cut the story short, I had to accept that there is nothing I can do with the networking issues and there are likely to be significant “bursts” of dropped pa
by tyingq 8y ago
"To cut the story short, I had to accept that there is nothing I can do with the networking issues and there are likely to be significant “bursts” of dropped packets that will cause disconnection of our MySQL8 cluster nodes.
So we know there is very likely an infrastructure problem. The question is, can we mitigate it on the application layer?"
That seems unwise. It's on DigitalOcean, so they could have migrated to different VPS instances and checked to see if the problem followed.
- 420codebro 8y agoAlso he could consider adding in some fancy redundancy (SD-WAN, BGP, IP-SLA, whatever) to ensure that a single point of network I/O failure does not cascade to a split-brain scenario. Might reduce bandwidth or increase latency, but a total outage is pretty painful when you're taking about distributed/replicated databases.
- dc352 8y agoWe are still looking into it and I’m in touch with DO and I hope to learn more from their support. Unfortunately, their response time is currently around 2 days. I guess my point here was that Our database should be resilient to this kind of infra issues and ideally self-heal if these are transient events.
- tyingq 8y agoYes, the work to get the cluster resilient is terrific. Just suggesting moving away from the problematic VPS if a root cause can't be found.