9 ms·
nslookup > server a.ns.apple.com Default server: a.ns.apple.com Address: 2620:149:ae0::53#53 Default server: a.ns.apple.com Address: 17.253.
by Animats 5y ago
nslookup
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer.apple.com
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net",
which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.
More:
nslookup
> developer-cdn.apple.com.akadns.net
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer-cdn.apple.com.akadns.net
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point
"developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?
It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.
- variant 5y agoI think something with DNSSEC: https://puck.nether.net/pipermail/outages-discussion/2022-March/002286.html https://puck.nether.net/pipermail/outages-discussion/2022-Ma...
- joveian 5y agoI noticed a few weeks ago that developer.apple.com was failing DNSSEC and that this had been going on for a while (follow the "previous analysis" links to see earlier errors as well): https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/ https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/ It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing. lima has an anaylsis of the issue causing trouble: https://news.ycombinator.com/item?id=30757487 https://news.ycombinator.com/item?id=30757487
- tptacek 5y agoAPPLE.COM isn't signed at all; this isn't a DNSSEC issue. In the future, if you want to check if something is DNSSEC-signed (things rarely are: DNSSEC is overwhelmingly not enabled on the commercial Internet), you can just `host -t ds <domain>`.
- joveian 5y agoI noticed it because developer.apple.com failed validation using systemd-resolved with DNSSEC enabled when someone posted a link on HN (but worked fine with DNSSEC disabled). It still does. The main apple site doesn't have that issue (the post I linked to gave the general, non-DNSSEC related issue this time). I tried several local utilities and options but couldn't find a reliable way to determine if a site would resolve under systemd-resolved with DNSSEC enabled other than using systemd-resolve with DNSSEC enabled. It seemed like any time dnsviz.net shows an error the domain will not resolve, but some things it shows as warnings also cause sites to not resolve while other warnings do not. My favorite is that Verisign's DNSSEC validator's domain fails to resolve with DNSSEC enabled. Possibly some or all of this is systemd-resolved doing the wrong thing, however the errors and warnings on dnsviz.net make me think this is not the case. www.google.com, for example, does not show any warnings or errors.
- tptacek 5y agoGOOGLE.COM is also not DNSSEC-signed. Seriously, almost nothing is.
- joveian 5y agoRight, but my point is "not DNSSEC-signed" does not seem to be the same as "free of configuration errors that prevent resolution of the name with DNSSEC enabled".
- tptacek 5y agoWhich configuration errors would those be? Without a DS record, there's no DNSSEC happening at the resolver, is there?
- tptacek 5y agoAAPLIMG.COM isn't DNSSEC-signed either.
- variant 5y agoYeah. Was wondering if they'd enabled it and backed off when it didn't work.
- frays 5y agoWorks with other DNS servers. $ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com Server: a.ns.apple.com Address: 17.253.200.1#53 ** server can't find developer-cdn.apple.com.akadns.net: REFUSED $ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1 Server: 1.1.1.1 Address: 1.1.1.1#53 Non-authoritative answer: developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com. Name: world-gen.g.aaplimg.com Address: 17.253.121.201 Name: world-gen.g.aaplimg.com Address: 17.253.121.202
- pul 5y agoNow also works on their authoritative DNS servers again: https://www.nslookup.io/dns-records/apple.com#authoritative https://www.nslookup.io/dns-records/apple.com#authoritative
- silisili 5y ago> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server. It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server. The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.
- jonfw 5y agoThis looks like an Akamai DNS load balancing solution. It will route a user to an endpoint based on a bunch of statistics (think location, availability, latency, and/or load), and will often handle caching and DDOS protection as well
- lima 5y ago> Or am I misreading that. Yes: developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net. developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com. world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com. apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com. apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com. apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com. apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com. The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com. They fixed that and now it's back up. This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB). > It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult. Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully. [1]: https://ns1.com/blog/glue-records-and-dedicated-dns https://ns1.com/blog/glue-records-and-dedicated-dns
- mnd999 5y agoCan we refer to this as “Doing a Facebook?”
- Karrot_Kream 5y agoThis has nothing to do with the BGP failures that FB had earlier. This is a DNS configuration problem. It's much simpler to fix.