3 ms·
> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clie
by throwaway09223 4y ago
> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds"
Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes.
DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design.
Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.
- mort96 4y ago> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way. But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?
- Serow225 4y agovia the magic of Anycast :)
- throwaway09223 4y agoYou sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example. DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.
- silisili 4y agoYes, with BGP and anycast. What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.