The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could

The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could do was accept that we were going to impact customers and that we just can't realistically account for misbehaving DNS infrastructure. We had something like a 5 minute TTL on our public record, and would wait 15 minutes after removing the IP from the DNS records before we would take action that would potentially break connectivity. By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control. What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.