4 ms·
I like to think of the problem as an inherent issue with cache invalidation in distributed systems that has no straightforward solution in an Internet architect
by devonkim 11y ago
I like to think of the problem as an inherent issue with cache invalidation in distributed systems that has no straightforward solution in an Internet architecture. DNS problems greatly resemble what it's like to have dangling pointers, for another analogy.
I think the expected solution approach is a little like validating 2FA logins. To account for system variability, you accept that transactions can take up to a certain time and you allow multiple values (the similarities probably end there, admittedly). With a more advanced solution you would even advise clients to invalidate their own cache too (similar to load shedding by advising / redirecting clients - you can see this in 2FA logins sometimes where you're asked for the sequence after the once-valid key just entered). So I think you'd need to accept multiple prior CNAME resolutions to account for longer lived transactions and make sure each entry change will be valid only for so long. Being able to be notified of these changes programmatically would be really nice though specifically for AWS. Perhaps AWS Lambda or SNS could be leveraged for pushing AWS DNS state change notifications to your system?
One approach I've seen some folks do is to simply reload their nginx configurations across their backend nodes to refresh the cached entry. It's probably intractable without sacrificing a lot of theoretical availability with in a degenerate state of reloading upon each request when your CNAME changes several times a second. Some of my colleagues have experienced problems even with the nginx proxy_pass directive that most people say is the most recommended free solution and also recommended in the article.