5 ms·

As someone who has recently dealt with a major government agency, the requirement of DNSSEC means that agencies can’t use Route53 as it doesn’t support DNSSEC.

As someone who has recently dealt with a major government agency, the requirement of DNSSEC means that agencies can’t use Route53 as it doesn’t support DNSSEC. This makes it much less likely that an agency will use something like Terraform for managing DNS, which would help in making sure all records are validated and easily auditable. Instead, many DNS records are managed by hand or with bespoke systems. The best case is that it’s managed by maybe Akamai, but records are still updated via support tickets and implemented by hand (as akamai’s Terraform support is pretty bad). So it’s relevent to mention DNSSEC because it’s spoken of within the government as a method of securing DNS, but in reality it makes things worse. It doesn’t protect against attacks like these and the end result is you are forced to use a crappy system that makes you even more vulnerable to these sorts of attacks. It’s the worst of all worlds. (Also, I was on the call they held for CIOs and tech leaders of all agencies, and it was astonishing how many of the questions in their chat made it obvious that many of the entrenched players didn’t even understand how this attack worked.)

minaguib 8y ago
Tangential, but since you mentioned Akamai. I wrote a framework at work that maintains authoritative DNS configuration truth (static, and advanced GSLB variations) as text files going through git and regular PRs, auditing, etc. The truth is then flushed out via APIs to multiple DNS providers (active-active), Akamai being one of them. Nobody's APIs are perfect, but I don't recall it being too difficult to push the state to their system.
- anoncontractor 8y ago
  For sure, you can totally make something like this happen, but again that means writing and securing a bespoke system. If you’re a team managing your AWS infrastructure with Terraform, it sucks to have to write a totally different system (and all the secrets management that goes along with it). Additionally, for a government agency you could have dozens of contractor teams, and you need a method of securing access for all of them. It’d be much easier to just use Route53 and delegated subdomains, as all teams are largely deployed and managing infra (with MFA) there. But it’s not allowed, because DNSSEC.
  - deleted 8y ago
    [deleted]
  - jrochkind1 8y ago
    Is there anything that would make it especially challenging for Route53 to support DNSSEC, or have they just not chosen to implement it?
    - anoncontractor 8y ago
      I don’t work for AWS, but I’ve talked to people that work on Route53. Amazon just isn’t willing to support it, none of their large customers care about it enough it seems like. Plus DNSSEC has all sorts of issues, some of which people in the thread have mentioned. Actually, tptacek has a blog post that summarizes a lot of what’s wrong with DNSSEC iirc.
      - getoffyour 8y ago
        This unwillingness may indicate a lack of competency, complacency as a market leader, or complicity with censoring regimes. How f difficult is it to sign a zone? DNSSEC isn't perfect (indeed it only provides assurances of record integrity and doesn't secure the channel); but it's certainly better than nothing, than no signature at all. If route53 can't or won't or doesn't have to because they don't want to implement DNSSEC, route53 is not suitable for .gov and .mil domains. It's really that simple. I get that you want to use terraform; I don't see why you think route53 is the only DNS that terraform works with.
amluto 8y ago
I wonder why Route53, etc. don’t support DNSSEC. Even without widespread client support for DNSSEC (and the AD bit doesn’t cut it), it would be quite nice if CA/B rules required that CAs enforce DNSSEC (if enabled for the domain) when checking CAA records. This would impose no burden at all on web browsers and only a minimal burden on CAs. Unfortunately, since Route53 and the like mostly don’t support DNSSEC, getting the benefit for domain owners is a pain in the neck. RFC 6844 seems to recommend but not require this validation. I don’t know whether newer rules require it. edit: Unsurprisingly, people are working on this: https://tools.ietf.org/html/draft-ietf-acme-caa-06 https://tools.ietf.org/html/draft-ietf-acme-caa-06, section 5.6, for example.
- zrm 8y ago
  One of the problems with DNSSEC is that it causes the size of DNS responses to increase substantially, which becomes a DDoS vector. The response to this is nominally to use response rate limiting, but the more people who use DNSSEC the less effective that is because it means more DNS servers the attacker can use to reflect their attack through without reaching the rate limit for any one. (RRL is also not ideal because a large DNS cache can hit the rate limit by just making a large number of legitimate queries.) And the more servers that use DNSSEC the harder to block a DDoS attack because it comes from a larger number of unique addresses. And now TXT records used for domain validation are becoming quite large as well. You add a ~hundred byte TXT record for each service you want to validate your domain against, but they all want the entry to go in the same place (at the root), so you do the DNS TXT query for the root and you get a big response. Adding DNSSEC on top of that might even blow the practical limit for UDP EDNS0 responses sometimes, never mind the DDoS potential. There may be a solution for that one -- specify a location for each TXT record instead of everyone using the root, so Facebook uses _facebook-domain-verification.[example.com], Google uses _google-site-verification.[example.com], etc. Then you wouldn't get a dozen large TXT records in response to a single query because they're each at a different name. But that's not currently what companies are doing, and a purist wouldn't like it because in principle controlling _google-site-verification.example.com doesn't mean you control all of example.com.
  - amluto 8y ago
belorn 8y ago
The article/directive specifically called out DNS account passwords on DNS hosting providers. If they were using support tickets and manually implemented changes by hand then there would not be an account. With no account for a control panel you don't have any risk that the email and password gets dumped on pastebin. In the directive it even speaks about password managers, which again implies that the issue they are talking about are DNS control panels like the one at Route53, likely with reused password, and not bespoke systems. For high priority domain names I would go so far and recommend against having a DNS control panel accounts on things like Route53. It has access control management but unless the department has the experience and skills it will likely end up with a single account that has full access with little oversight when something goes wrong. Those departments are likely better served going through a domain management company that handles all the security and manage who can request changes and how. If they have the experience and skills then managing it fully themselves can be a better option, but then what they really need is a registrar which allows for DNSSEC keys to be forwarded (with for example a cds record).
- bdhess 8y ago
  > The article/directive specifically called out DNS account passwords on DNS hosting providers. If they were using support tickets and manually implemented changes by hand then there would not be an account. I don't think it's difficult to imagine that, somewhere within the massive federal bureaucracy, there is a team that works on DNS record tickets filed by other parts of the bureaucracy, and logs into a web console with a DNS hosting provider in order to fulfill those tickets.
  - anoncontractor 8y ago
    Yes, this is pretty close to how this agency does things. Your contracting team files a support request for a record change and a manager from within the agency approves. The contractor that owns DNS changes then goes and does it. This makes all sorts of other things bad, like generating TLS certs with automation, as typically a validation for cert generation will be done via a DNS entry to prove ownership. But that requires a support ticket, so no automation will work here.
numbsafari 8y ago
Just stopped by to mention that Google’s Cloud DNS supports DNSSEC. You should be able to configure it via Terraform. Not an endorsement of DNSSEC, just that it’s not impossible to support it.
- deleted 8y ago
  [deleted]

minaguib 8y ago

Tangential, but since you mentioned Akamai. I wrote a framework at work that maintains authoritative DNS configuration truth (static, and advanced GSLB variations) as text files going through git and regular PRs, auditing, etc. The truth is then flushed out via APIs to multiple DNS providers (active-active), Akamai being one of them. Nobody's APIs are perfect, but I don't recall it being too difficult to push the state to their system.

anoncontractor 8y ago
For sure, you can totally make something like this happen, but again that means writing and securing a bespoke system. If you’re a team managing your AWS infrastructure with Terraform, it sucks to have to write a totally different system (and all the secrets management that goes along with it). Additionally, for a government agency you could have dozens of contractor teams, and you need a method of securing access for all of them. It’d be much easier to just use Route53 and delegated subdomains, as all teams are largely deployed and managing infra (with MFA) there. But it’s not allowed, because DNSSEC.
- deleted 8y ago
  [deleted]
- jrochkind1 8y ago
  Is there anything that would make it especially challenging for Route53 to support DNSSEC, or have they just not chosen to implement it?
  - anoncontractor 8y ago
    I don’t work for AWS, but I’ve talked to people that work on Route53. Amazon just isn’t willing to support it, none of their large customers care about it enough it seems like. Plus DNSSEC has all sorts of issues, some of which people in the thread have mentioned. Actually, tptacek has a blog post that summarizes a lot of what’s wrong with DNSSEC iirc.
    - getoffyour 8y ago
      This unwillingness may indicate a lack of competency, complacency as a market leader, or complicity with censoring regimes. How f difficult is it to sign a zone? DNSSEC isn't perfect (indeed it only provides assurances of record integrity and doesn't secure the channel); but it's certainly better than nothing, than no signature at all. If route53 can't or won't or doesn't have to because they don't want to implement DNSSEC, route53 is not suitable for .gov and .mil domains. It's really that simple. I get that you want to use terraform; I don't see why you think route53 is the only DNS that terraform works with.

amluto 8y ago

I wonder why Route53, etc. don’t support DNSSEC. Even without widespread client support for DNSSEC (and the AD bit doesn’t cut it), it would be quite nice if CA/B rules required that CAs enforce DNSSEC (if enabled for the domain) when checking CAA records. This would impose no burden at all on web browsers and only a minimal burden on CAs. Unfortunately, since Route53 and the like mostly don’t support DNSSEC, getting the benefit for domain owners is a pain in the neck. RFC 6844 seems to recommend but not require this validation. I don’t know whether newer rules require it. edit: Unsurprisingly, people are working on this: https://tools.ietf.org/html/draft-ietf-acme-caa-06 https://tools.ietf.org/html/draft-ietf-acme-caa-06, section 5.6, for example.

zrm 8y ago
One of the problems with DNSSEC is that it causes the size of DNS responses to increase substantially, which becomes a DDoS vector. The response to this is nominally to use response rate limiting, but the more people who use DNSSEC the less effective that is because it means more DNS servers the attacker can use to reflect their attack through without reaching the rate limit for any one. (RRL is also not ideal because a large DNS cache can hit the rate limit by just making a large number of legitimate queries.) And the more servers that use DNSSEC the harder to block a DDoS attack because it comes from a larger number of unique addresses. And now TXT records used for domain validation are becoming quite large as well. You add a ~hundred byte TXT record for each service you want to validate your domain against, but they all want the entry to go in the same place (at the root), so you do the DNS TXT query for the root and you get a big response. Adding DNSSEC on top of that might even blow the practical limit for UDP EDNS0 responses sometimes, never mind the DDoS potential. There may be a solution for that one -- specify a location for each TXT record instead of everyone using the root, so Facebook uses _facebook-domain-verification.[example.com], Google uses _google-site-verification.[example.com], etc. Then you wouldn't get a dozen large TXT records in response to a single query because they're each at a different name. But that's not currently what companies are doing, and a purist wouldn't like it because in principle controlling _google-site-verification.example.com doesn't mean you control all of example.com.
- amluto 8y ago

belorn 8y ago

The article/directive specifically called out DNS account passwords on DNS hosting providers. If they were using support tickets and manually implemented changes by hand then there would not be an account. With no account for a control panel you don't have any risk that the email and password gets dumped on pastebin. In the directive it even speaks about password managers, which again implies that the issue they are talking about are DNS control panels like the one at Route53, likely with reused password, and not bespoke systems. For high priority domain names I would go so far and recommend against having a DNS control panel accounts on things like Route53. It has access control management but unless the department has the experience and skills it will likely end up with a single account that has full access with little oversight when something goes wrong. Those departments are likely better served going through a domain management company that handles all the security and manage who can request changes and how. If they have the experience and skills then managing it fully themselves can be a better option, but then what they really need is a registrar which allows for DNSSEC keys to be forwarded (with for example a cds record).

bdhess 8y ago
> The article/directive specifically called out DNS account passwords on DNS hosting providers. If they were using support tickets and manually implemented changes by hand then there would not be an account. I don't think it's difficult to imagine that, somewhere within the massive federal bureaucracy, there is a team that works on DNS record tickets filed by other parts of the bureaucracy, and logs into a web console with a DNS hosting provider in order to fulfill those tickets.
- anoncontractor 8y ago
  Yes, this is pretty close to how this agency does things. Your contracting team files a support request for a record change and a manager from within the agency approves. The contractor that owns DNS changes then goes and does it. This makes all sorts of other things bad, like generating TLS certs with automation, as typically a validation for cert generation will be done via a DNS entry to prove ownership. But that requires a support ticket, so no automation will work here.

numbsafari 8y ago

Just stopped by to mention that Google’s Cloud DNS supports DNSSEC. You should be able to configure it via Terraform. Not an endorsement of DNSSEC, just that it’s not impossible to support it.

deleted 8y ago
[deleted]