11 ms·

HBO Now DNSSEC Misconfiguration Makes Site Unavailable from Comcast Networks

81 points by danyork 12y ago

wtbob 12y ago
Also unavailable from Android devices…
- danyork 12y ago
  Which would make sense because they may be set to use Google's Public DNS Servers (8.8.8.8 and 8.8.4.4 and the IPv6 equivalents). Google Public DNS has been performing full DNSSEC validation since May 2013: http://www.internetsociety.org/deploy360/blog/2013/05/confirmed-googles-public-dns-now-performs-dnssec-validation-for-all-queries-by-default/ http://www.internetsociety.org/deploy360/blog/2013/05/confir...
  - yuhong 12y ago
    Yea, it is fortunate that people on Twitter quickly figured this out.
    - rb12345 12y ago
      To be fair though, it's not that hard to test for DNSSEC breakage if you have dig available. If "dig +cdflag example.com A IN" works and "dig +adflag example.com A IN" returns SERVFAIL, it's DNSSEC's fault. (If that latter command returns responses without the ad flag, the domain lacks DNSSEC signing.)
- danyork 12y ago
  If it's currently unavailable on Android, you should be able to start seeing the site again whenever the TTL of the DNS records ages out... this just happened for me on my own home network where I have a DNSSEC-validating DNS resolver on the edge of my network. However, when I do "dig +dnssec @8.8.8.8 ds hbonow.com" I don't get any DS records back, indicating that the issue should be fixed on Google's PDNS. Can you tell what DNS servers you are using on your Android device? It may be that your ISP is doing DNSSEC validation and experiencing a similar TTL issue.
  - wtbob 12y ago
    I was being snarky, since HBO Now reportedly supports only Apple clients.
    - danyork 12y ago
      Too funny... completely missed the snark in the midst of all the discussion here. :-) And you're right - HBO Now is only on Apple devices right now... so getting to the website from Android wouldn't matter.
tptacek 12y ago
This shouldn't be possible. In a sane design, a transient policy failure should produce a warning light or a browser popup; something that applications can use to sensibly convey what happened, in some form, to users. Instead, because DNSSEC makes security policy decisions at a very low layer, in a protocol that virtually all applications expect to work transparently so long as you have connectivity, you get this: instead of a browser popup, a connectivity outage so complete that it is indistinguishable to users from "the site you are trying to get to doesn't exist and never has". DNSSEC is a terrible, terrible idea. Comcast should stop validating it, immediately. If someone from Comcast's network engineering team is reading this: I was a network engineer, and then worked for 4 years with tier-1 engineering getting Netflow monitoring deployed and scalable. I've been writing DNS security software since I was 19 years old, in the 1990s. You have reasons for whatever configuration you've decided on, but I will take as much time out of my day as is productive for you to convince you to stop working on rolling this out. What happened yesterday with HBO Now, which had half of Twitter screaming "NET NEUTRALITY OHNOZ!", is going to keep happening over and over again. See also: https://news.ycombinator.com/item?id=8894902 https://news.ycombinator.com/item?id=8894902
- Diederich 12y ago
  Thomas, I'll ask directly even though I believe I know the answer. But this topic is relatively dense so there's a non-trivial chance I've got some wires crossed. In short: with total objectivity, what are all of the possible, theoretical advantages to the people using order.hbonow.com of HBO using DNSSEC + https + (maybe) HSTS rather than them ONLY using https + HSTS? I'm posting this (I think) leading question so that it might be possible to cut through VOLUMES of related discussion, and see a relatively simple, straightforward answer. Thank you! PS: I have read through your Against DNSSEC post, and the large amounts of related discussion, but I'm still seeking a technical yet succinct response that I can easily 'carry around' with me.
  - tptacek 12y ago
    There is zero reason for HBO to be using DNSSEC today. Browsers don't support it, and email is insecure with or without it, and that describes pretty much their whole attack surface. It was a totally unforced error, which is presumably why HBO's response to this debacle was to eliminate DNSSEC.
danyork 12y ago
Several people have confirmed to me that they were unable to see the site from Google's Public DNS Servers (8.8.8.8, 8.8.4.4 and the IPv6 addresses). This makes sense given that Google has been performing DNSSEC validation since early 2013. I've updated the post to more clearly note the fact that this was not just an issue on Comcast networks. (I mentioned that it would be an issue on any network performing DNSSEC validation, but then only gave Comcast as an example.)
st3fan 12y ago
"As a result, the many networks around the world that perform DNSSEC validation to ensure that customers are getting to the correct sites (versus being redirected to bogus sites for phishing or malware) were blocking customers from getting to the possibly bogus order.hbonow.com!" The "versus being redirected to bogus sites for phishing or malware" part here is funny. Because that generally happens when some scammer registeres hb0now.com. Not when someone is intercepting your DNS. And actually, in case of the latter, all bets are off anyways. Because client resolvers DO NOT CHECK DNSSEC. (Yes, I know you are on your custom configured Linux box or OpenBSD firewall and that DNSSEC works wonderful for you. But the majority of the internet using world with OS X or Windows behind a $15 DSL router or WiFi AP does not.) Stop. The. DNSSEC. Nonsense.
- xorcist 12y ago
  There have been many instances of unknown entities intercepting DNS which went unnoticed for quite some time. Hence DNSSEC.
- danyork 12y ago
  And actually... > And actually, in case of the latter, all bets are off anyways. Because client resolvers DO NOT CHECK DNSSEC. ... please visit APNIC's DNSSEC Statistics site where you will see that about 12% of all DNS queries globally ARE being validated by DNSSEC: http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g=0 http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g... In Sweden this is about 71%: http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1 http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1 Slovenia 67%, Estonia 55%, Denmark 48% ... on down to the USA at 23%: http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1 http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1 So DNSSEC validation very definitely * IS * happening out there!
  - tptacek 12y ago
    This is "checking" in the sense that "everyone at Comcast was 'checking' when HBO Now broke".
  - icebraining 12y ago
Panino 12y ago
DNSSEC causes more problems than it solves. And even if it were reliable, it's still based on obsolete crypto and isn't even encrypted. But on top of that, it's an outage factory.
- samplonius 12y ago
  Why would it make sense to encrypt queries and replies from a public data source? Signing the data is all that is needed.
  - jessaustin 12y ago
    Some users might not want intermediate hosts to see a request for the IP addr of "ginger-midget-porn.com", for example.

81 points by danyork 12y ago

wtbob 12y ago

Also unavailable from Android devices…

danyork 12y ago
Which would make sense because they may be set to use Google's Public DNS Servers (8.8.8.8 and 8.8.4.4 and the IPv6 equivalents). Google Public DNS has been performing full DNSSEC validation since May 2013: http://www.internetsociety.org/deploy360/blog/2013/05/confirmed-googles-public-dns-now-performs-dnssec-validation-for-all-queries-by-default/ http://www.internetsociety.org/deploy360/blog/2013/05/confir...
- yuhong 12y ago
  Yea, it is fortunate that people on Twitter quickly figured this out.
  - rb12345 12y ago
    To be fair though, it's not that hard to test for DNSSEC breakage if you have dig available. If "dig +cdflag example.com A IN" works and "dig +adflag example.com A IN" returns SERVFAIL, it's DNSSEC's fault. (If that latter command returns responses without the ad flag, the domain lacks DNSSEC signing.)
danyork 12y ago
If it's currently unavailable on Android, you should be able to start seeing the site again whenever the TTL of the DNS records ages out... this just happened for me on my own home network where I have a DNSSEC-validating DNS resolver on the edge of my network. However, when I do "dig +dnssec @8.8.8.8 ds hbonow.com" I don't get any DS records back, indicating that the issue should be fixed on Google's PDNS. Can you tell what DNS servers you are using on your Android device? It may be that your ISP is doing DNSSEC validation and experiencing a similar TTL issue.
- wtbob 12y ago
  I was being snarky, since HBO Now reportedly supports only Apple clients.
  - danyork 12y ago
    Too funny... completely missed the snark in the midst of all the discussion here. :-) And you're right - HBO Now is only on Apple devices right now... so getting to the website from Android wouldn't matter.

tptacek 12y ago

This shouldn't be possible. In a sane design, a transient policy failure should produce a warning light or a browser popup; something that applications can use to sensibly convey what happened, in some form, to users. Instead, because DNSSEC makes security policy decisions at a very low layer, in a protocol that virtually all applications expect to work transparently so long as you have connectivity, you get this: instead of a browser popup, a connectivity outage so complete that it is indistinguishable to users from "the site you are trying to get to doesn't exist and never has". DNSSEC is a terrible, terrible idea. Comcast should stop validating it, immediately. If someone from Comcast's network engineering team is reading this: I was a network engineer, and then worked for 4 years with tier-1 engineering getting Netflow monitoring deployed and scalable. I've been writing DNS security software since I was 19 years old, in the 1990s. You have reasons for whatever configuration you've decided on, but I will take as much time out of my day as is productive for you to convince you to stop working on rolling this out. What happened yesterday with HBO Now, which had half of Twitter screaming "NET NEUTRALITY OHNOZ!", is going to keep happening over and over again. See also: https://news.ycombinator.com/item?id=8894902 https://news.ycombinator.com/item?id=8894902

Diederich 12y ago
Thomas, I'll ask directly even though I believe I know the answer. But this topic is relatively dense so there's a non-trivial chance I've got some wires crossed. In short: with total objectivity, what are all of the possible, theoretical advantages to the people using order.hbonow.com of HBO using DNSSEC + https + (maybe) HSTS rather than them ONLY using https + HSTS? I'm posting this (I think) leading question so that it might be possible to cut through VOLUMES of related discussion, and see a relatively simple, straightforward answer. Thank you! PS: I have read through your Against DNSSEC post, and the large amounts of related discussion, but I'm still seeking a technical yet succinct response that I can easily 'carry around' with me.
- tptacek 12y ago
  There is zero reason for HBO to be using DNSSEC today. Browsers don't support it, and email is insecure with or without it, and that describes pretty much their whole attack surface. It was a totally unforced error, which is presumably why HBO's response to this debacle was to eliminate DNSSEC.

danyork 12y ago

Several people have confirmed to me that they were unable to see the site from Google's Public DNS Servers (8.8.8.8, 8.8.4.4 and the IPv6 addresses). This makes sense given that Google has been performing DNSSEC validation since early 2013. I've updated the post to more clearly note the fact that this was not just an issue on Comcast networks. (I mentioned that it would be an issue on any network performing DNSSEC validation, but then only gave Comcast as an example.)

st3fan 12y ago

"As a result, the many networks around the world that perform DNSSEC validation to ensure that customers are getting to the correct sites (versus being redirected to bogus sites for phishing or malware) were blocking customers from getting to the possibly bogus order.hbonow.com!" The "versus being redirected to bogus sites for phishing or malware" part here is funny. Because that generally happens when some scammer registeres hb0now.com. Not when someone is intercepting your DNS. And actually, in case of the latter, all bets are off anyways. Because client resolvers DO NOT CHECK DNSSEC. (Yes, I know you are on your custom configured Linux box or OpenBSD firewall and that DNSSEC works wonderful for you. But the majority of the internet using world with OS X or Windows behind a $15 DSL router or WiFi AP does not.) Stop. The. DNSSEC. Nonsense.

xorcist 12y ago
There have been many instances of unknown entities intercepting DNS which went unnoticed for quite some time. Hence DNSSEC.
danyork 12y ago
And actually... > And actually, in case of the latter, all bets are off anyways. Because client resolvers DO NOT CHECK DNSSEC. ... please visit APNIC's DNSSEC Statistics site where you will see that about 12% of all DNS queries globally ARE being validated by DNSSEC: http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g=0 http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g... In Sweden this is about 71%: http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1 http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1 Slovenia 67%, Estonia 55%, Denmark 48% ... on down to the USA at 23%: http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1 http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1 So DNSSEC validation very definitely * IS * happening out there!
- tptacek 12y ago
  This is "checking" in the sense that "everyone at Comcast was 'checking' when HBO Now broke".
- icebraining 12y ago

Panino 12y ago

DNSSEC causes more problems than it solves. And even if it were reliable, it's still based on obsolete crypto and isn't even encrypted. But on top of that, it's an outage factory.

samplonius 12y ago
Why would it make sense to encrypt queries and replies from a public data source? Signing the data is all that is needed.
- jessaustin 12y ago
  Some users might not want intermediate hosts to see a request for the IP addr of "ginger-midget-porn.com", for example.