4 ms·

Absolutely amazing how many billion $+ companies are single homed for DNS. I wonder how much they spend on multi-AZ redundant architectures...

orblivion 5y ago
So here's a weird question: Supposing companies multi-home for DNS, or whatever other essential service, via multiple service providers. Whatever multi-home means, why can't there just be one service provider that does that? And are we sure that these service providers aren't already doing that as best we might hope for? (For instance, Amazon already has multiple zones, etc.) I suppose the one thing this can't protect against is some sort of political (broadly defined) threat related to the company itself.
- lxgr 5y ago
  > Whatever multi-home means, why can't there just be one service provider that does that? Many of these outages are due to pushing broken artifacts or configuration to production. A single provider can pretty easily offer geographic or network topological redundancy, but administrative and/or technological independence is pretty hard to achieve in a single company.
  - knute 5y ago
    I believe EasyDNS can automatically push DNS settings to Route53 to host DNS in AWS. Doesn't protect you from fat-fingering a change, but you should be resilient to either EasyDNS or Route53 going down. https://kb.easydns.com/knowledge/easyroute53/ https://kb.easydns.com/knowledge/easyroute53/
  - orblivion 5y ago
    I mean, I guess what I'm saying is that in theory a single provider could purposely keep two different departments that manage their own artifacts independently.
    - topranks 5y ago
      Records have to be kept in sync. If one dept deletes a record and the other doesn’t, how do you decide who’s right? You could add a third dept that gives them both orders, but now that third dept is a single point of failure.
      - orblivion 5y ago
        If I were a customer of two different companies for the sake of redundancy, wouldn't I have that same challenge? I could be my own point of failure. Though, I suppose if I'm responsible for it, I fix it faster for myself.
        JetSpiegel 5y ago
        In this particular case, the Akamai clients did not push broken artifacts, so sounds like at least this particular instance would be avoided.
nexuist 5y ago
Might be survivorship bias. Multi-AZ arch protects against all other failures, so the only one that remains visible to the outside world is DNS.
zxcvbn4038 5y ago
Most CDNs offer huge incentives for sending them more traffic, a lot of time you end up in a contract obligated to handle X requests and Y gigabytes of traffic per month. But personally I believe you should never have a single provider for anything - particularly when it’s acceptable for a company to cut you off with no warning or recourse.
toast0 5y ago
Using multiple providers for mostly static DNS is easy, pick one as primary and AXFR to the other and notifications and whatever. Or it's not too hard to keep a zone file in source control and sync it to the providers. Using multiple providers for fancy DNS, like only providing IPs that pass healthchecks or geotargetting users to datacenters gets pretty hard, because the different providers have similar capabilities, but no uniform interface, so you've either got to do it manually, or you have to build out your own abstraction that is probably limiting. If possible, insourcing DNS makes the most sense to me, because if you can't keep your service online, it's not the worst if your DNS is offline; and if you can keep your service online, you probably won't mess up your DNS too badly.
- jfoutz 5y ago
  So much this. Keeping feature by feature parity is the tricky part.
topranks 5y ago
Problem is, if your on Akamai’s CDN, only Akamai know where the local caches are. You need to be on their DNS only.

orblivion 5y ago

So here's a weird question: Supposing companies multi-home for DNS, or whatever other essential service, via multiple service providers. Whatever multi-home means, why can't there just be one service provider that does that? And are we sure that these service providers aren't already doing that as best we might hope for? (For instance, Amazon already has multiple zones, etc.) I suppose the one thing this can't protect against is some sort of political (broadly defined) threat related to the company itself.

lxgr 5y ago
> Whatever multi-home means, why can't there just be one service provider that does that? Many of these outages are due to pushing broken artifacts or configuration to production. A single provider can pretty easily offer geographic or network topological redundancy, but administrative and/or technological independence is pretty hard to achieve in a single company.
- knute 5y ago
  I believe EasyDNS can automatically push DNS settings to Route53 to host DNS in AWS. Doesn't protect you from fat-fingering a change, but you should be resilient to either EasyDNS or Route53 going down. https://kb.easydns.com/knowledge/easyroute53/ https://kb.easydns.com/knowledge/easyroute53/
- orblivion 5y ago
  I mean, I guess what I'm saying is that in theory a single provider could purposely keep two different departments that manage their own artifacts independently.
  - topranks 5y ago
    Records have to be kept in sync. If one dept deletes a record and the other doesn’t, how do you decide who’s right? You could add a third dept that gives them both orders, but now that third dept is a single point of failure.
    - orblivion 5y ago
      If I were a customer of two different companies for the sake of redundancy, wouldn't I have that same challenge? I could be my own point of failure. Though, I suppose if I'm responsible for it, I fix it faster for myself.
      - JetSpiegel 5y ago
        In this particular case, the Akamai clients did not push broken artifacts, so sounds like at least this particular instance would be avoided.

nexuist 5y ago

Might be survivorship bias. Multi-AZ arch protects against all other failures, so the only one that remains visible to the outside world is DNS.

zxcvbn4038 5y ago

Most CDNs offer huge incentives for sending them more traffic, a lot of time you end up in a contract obligated to handle X requests and Y gigabytes of traffic per month. But personally I believe you should never have a single provider for anything - particularly when it’s acceptable for a company to cut you off with no warning or recourse.

toast0 5y ago

Using multiple providers for mostly static DNS is easy, pick one as primary and AXFR to the other and notifications and whatever. Or it's not too hard to keep a zone file in source control and sync it to the providers. Using multiple providers for fancy DNS, like only providing IPs that pass healthchecks or geotargetting users to datacenters gets pretty hard, because the different providers have similar capabilities, but no uniform interface, so you've either got to do it manually, or you have to build out your own abstraction that is probably limiting. If possible, insourcing DNS makes the most sense to me, because if you can't keep your service online, it's not the worst if your DNS is offline; and if you can keep your service online, you probably won't mess up your DNS too badly.

jfoutz 5y ago
So much this. Keeping feature by feature parity is the tricky part.

topranks 5y ago

Problem is, if your on Akamai’s CDN, only Akamai know where the local caches are. You need to be on their DNS only.