4 ms·

HE.net problem

78 points by greyface- 2y ago

bediger4000 2y ago
Is there some context that would aid in understanding this? Because it doesn't make sense.
- trvrprkr 2y ago
  Hurricane Electric (HE) provides, among other things, DNS services. When a domain is placed in `clientHold`, as has happened to HE due to a spurious phishing report, it causes the domain to no longer resolve. So the DNS records for HE and all of HE's customers are gradually becoming unresolvable as caches expire.
  - KennyBlanken 2y ago
    For anyone not well versed in DNS who needs an ELI5 level: This impacts anyone using HE.net for their domain name service because a domain name specifies its nameservers (in an NS record) using fully qualified domain names. I believe it's ns1.he.net and ns2.he.net. User at WildISP tries to go to a site. Resolving nameserver at WildISP.net fetches the NS records for samplewhamplesite.com from a root server and gets pointed to ns1.he.net...which no longer exists because the entire domain disappeared from the root servers. It looks like things are back up? Edit: nope, still in clienthold.
  - dave78 2y ago
    They offer a nice (free) secondary DNS service, which I use for all my domains. From my limited interaction with them, they seem like a great company. They also provide some nice IPv6 tutorials as well, and offer free IPv6 tunnels for those who can't get native IPv6. All in all they seem like a model company that believes in giving back to the internet community in meaningful ways. Unfortunately, I use them to host secondary DNS for all my domains, so this Network Solutions stupidity is hitting home for me right now. Then again, I had enough issues with NS back in the 1990s that as soon as that monopoly was broken I switched and never looked back. TBH I'm a little surprised that HE would use NS, but perhaps they did thinking that NS would provide proper enterprise-level support for a major backbone provider and not cut them off on a holiday with no recourse - behavior you might expect or fear from a cut-rate provider.
  - toast0 2y ago
    Edit: since the domain is fixed, this isn't reproducable anymore. An interesting thing is that if you have a .net domain with he.net nameservers, the .net authoritative servers will give you full glue records with the A/AAAA for nsX.he.net. But if you ask for he.net, you get back NXDOMAIN. Example domain removed; no longer relevant. NetworkSolutions is a truly terrible registrar, and anyone who still has a domain there should use this occasion to switch to somewhere better, which is almost anywhere. IMHO, it may be worthwhile for a real business where their domain is important to move to an expensive corporate registrar like MarkMonitor or CSC, and looking into registry lock, etc.
  - slyall 2y ago
    I guess this is one of the reasons AWS uses multiple domains in different TLDs for it's customer's name servers in route53[1] A provider going rouge or a domain expiring will probably still leave 3 perfectly working. [1] eg one of mine has awsdns-21.com, awsdns-50.co.uk, awsdns-11.net and awsdns-42.org
- LinuxBender 2y ago
  Looking to see what escalation numbers I may still have... [Edit]: Sent them what I have but it isn't much. Most human contacts have been replaced by menu's. I hope the last one I sent helps. It appears someone managed to get NetworkSolutions (now web.com) to put the domain he.net on clientHold for one Phishing complaint, unrelated to he.net meaning someone did a lookup on bgp.he.net and reported he.net instead of the domain they were looking up. If true it means web.com did not do any due diligence, investigation, etc... and just put a very popular domain with many shared services on hold. The domain he.net is used by MANY services and those services are likely going to start breaking. I have no idea what's going on over at web.com. As a side note, I recently transferred all my domains away from NetSol/web.com as they lost the ability to define custom name servers (glue in the root servers) which is bizarre to me, given many of the meme-style resellers of resellers can still do this. I had many other strange experiences with them, including having to teach their escalation contacts how to use dig outside of their own name servers.
  - deleted 2y ago
    [deleted]
    - LinuxBender 2y ago
      I was with Network Solutions from 1997 to 2023. They were bought by web.com in 2018. They keep the name on the old domain and don't redirect to web.com because they have some non-API automation in place on the old domain.
      - qual 2y ago
        Wikipedia has it incorrect then, as they list it as "formerly web.com" ("Network Solutions, LLC, formerly Web.com is an American-based technology company"). Thanks for the clarification!
  - fragmede 2y ago
    > As a side note, I recently transferred all my domains away from NetSol/web.com as they lost the ability to define custom name servers FFS is that what happened? They recently bought the registrar I use and a domain of mine with custom nameservers isn't working and I haven't had a chance to troubleshoot.
- KennyBlanken 2y ago
  https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en https://www.icann.org/resources/pages/epp-status-codes-2014-... > This status code tells your domain's registry to not activate your domain in the DNS and as a consequence, it will not resolve. It is an uncommon status that is usually enacted during legal disputes, non-payment, or when your domain is subject to deletion. > Often, this status indicates an issue with your domain that needs resolution. If so, you should contact your registrar to resolve the issue. If your domain does not have any issues, but you need it to resolve, you must first contact your registrar and request that they remove this status code.
speckx 2y ago
I can't believe that NetworkSolutions doesn't have a list of significant ISPs by domain name or something. I would proactively work with significant ISPs before suspending the entire domain and interrupting their services.
- bifrost 2y ago
  Its not really going to affect their services, maybe email in/out of it but thats about it. NetSol should've done better though, this makes them look like clowns.
  - a-dub 2y ago
    > Its not really going to affect their services, maybe email in/out of it but thats about it idk. how do you know that their management capability can run sans dns?
    - bifrost 2y ago
      Because they run their own DNS servers and inside of their network they're going to be using their own infra...
      - a-dub 2y ago
        idk. aren't they tier 1? what if it breaks peering? is all their management capability strictly "inside" their network? what would happen to the rest of the internet if they dropped off?
- KennyBlanken 2y ago
  There's a reason people were practically partying in the streets when Network Solutions lost their monopoly. The fact that their netops / support staff have no way to escalate an issue like this is the bigger issue. IMHO the giant ISP shouldn't get a better response time or procedure; everyone should be able to get help in an emergency like this. Really, NANOG have themselves to blame. Half the reason NANOG was started was to help high level techies bypass low-level operations/support staff. Everyone should have worked together to make sure their own orgs could appropriately handle calls/issues like this. Instead, with the issue mostly fixed for them, they stopped caring, and all the peons and techies who don't work for Fortune 500 or famous tech companies are still fucked...
killerstorm 2y ago
[flagged]
- Reviving1514 2y ago
  Please elaborate, how would block chain solve this problem?
  - bifrost 2y ago
    It would not.
    - deleted 2y ago
      [deleted]
  - fragmede 2y ago
    If we treat Blockchain as a database with specific properties, and use that to back DNS, then we have a DNS system with those specific properties, instead of the system we currently have, which has different specific properties. Some of those properties are desirable, but as we see here, some of them are not.
  - killerstorm 2y ago
    There's a number of DNS-like protocols on blockchain, e.g. ENS (Ethereum Name System - https://ens.domains/ https://ens.domains/), Handshake (https://handshake.org/ https://handshake.org/). They provide same functionality as DNS[sec]. You are not at whim of a corpo, nobody can suspend you, corrupting or hijacking records is pretty much impossible...
    - warkdarrior 2y ago
      That sounds like an excellent feature. Especially for phishing and other scammers who need bulletproof hosting.
      - killerstorm 2y ago
deleted 2y ago
[deleted]
codetrotter 2y ago
Status in whois response is “active” for me, and he.net resolves from DNS for me, so I guess it’s been fixed? But why did they list a @he.net email address as one of the emails to contact them on? Wasn’t the problem that he.net would not resolve because of that very client hold?
- deleted 2y ago
  [deleted]
- KennyBlanken 2y ago
  Confirmed at 3:30 CT
  - codetrotter 2y ago
    I read a bit more in the linked email thread. Some other message said that caching would likely make it still resolve in most places and that checking the root servers would show the problem and that elsewhere it would show up around 48 hours after, when the caches expire. Still not sure if the status it shows in Whois means that it’s resolved or not.
  - LinuxBender 2y ago
    Indeed was Domain Status: clientHold https://icann.org/epp#clientHold https://icann.org/epp#clientHold now Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited https://icann.org/epp#clientTransferProhibited Verified he.net is correctly listed in the root servers, though I did not check prior. dig +nocookie @i.gtld-servers.net he.net ;; AUTHORITY SECTION: he.net. 172800 IN NS ns1.he.net. he.net. 172800 IN NS ns2.he.net. he.net. 172800 IN NS ns3.he.net. he.net. 172800 IN NS ns4.he.net. he.net. 172800 IN NS ns5.he.net. ;; ADDITIONAL SECTION: ns1.he.net. 172800 IN AAAA 2001:470:100::2 ns1.he.net. 172800 IN A 216.218.130.2 ns2.he.net. 172800 IN AAAA 2001:470:200::2 ns2.he.net. 172800 IN A 216.218.131.2 ns3.he.net. 172800 IN AAAA 2001:470:300::2 ns3.he.net. 172800 IN A 216.218.132.2 ns4.he.net. 172800 IN AAAA 2001:470:400::2 ns4.he.net. 172800 IN A 216.66.1.2 ns5.he.net. 172800 IN AAAA 2001:470:500::2 ns5.he.net. 172800 IN A 216.66.80.18 which belong to Hurricane [1] [1] - https://bgp.he.net/AS6939#_whois https://bgp.he.net/AS6939#_whois
deleted 2y ago
[deleted]
basilgohar 2y ago
First off, Hurricane Electric has always been one of the "cool" companies on the Internet since the early days (of public Internet, at least). So, my goodwill with them tells me that Network Solutions, a company that most certainly has NOT been one of the cool companies, is at fault. I feel and know this to be the case now, as well. Having said that, I am also greatly pleased that the NANOG mailing list is public and plebes like us can see these kinds of discussions, which I feel are becoming harder to find as time goes on as services get consolidated into larger, government-friendly corporate private digital gardens.
- deleted 2y ago
  [deleted]
byyll 2y ago
Great news! It's a private company and they can do whatever they want! Just like HE likes to censor.
omoikane 2y ago
The hold appears to have been removed as of few minutes ago: https://mailman.nanog.org/pipermail/nanog/2024-July/225903.html https://mailman.nanog.org/pipermail/nanog/2024-July/225903.h... I didn't know HE.net was using Network Solutions, maybe they will switch to someone else soon. My personal experience with Network Solutions wasn't so great (lots of trouble and human intervention needed to transfer a domain out of their service).

HE.net problem

78 points by greyface- 2y ago

bediger4000 2y ago

Is there some context that would aid in understanding this? Because it doesn't make sense.

trvrprkr 2y ago
Hurricane Electric (HE) provides, among other things, DNS services. When a domain is placed in `clientHold`, as has happened to HE due to a spurious phishing report, it causes the domain to no longer resolve. So the DNS records for HE and all of HE's customers are gradually becoming unresolvable as caches expire.
- KennyBlanken 2y ago
  For anyone not well versed in DNS who needs an ELI5 level: This impacts anyone using HE.net for their domain name service because a domain name specifies its nameservers (in an NS record) using fully qualified domain names. I believe it's ns1.he.net and ns2.he.net. User at WildISP tries to go to a site. Resolving nameserver at WildISP.net fetches the NS records for samplewhamplesite.com from a root server and gets pointed to ns1.he.net...which no longer exists because the entire domain disappeared from the root servers. It looks like things are back up? Edit: nope, still in clienthold.
- dave78 2y ago
  They offer a nice (free) secondary DNS service, which I use for all my domains. From my limited interaction with them, they seem like a great company. They also provide some nice IPv6 tutorials as well, and offer free IPv6 tunnels for those who can't get native IPv6. All in all they seem like a model company that believes in giving back to the internet community in meaningful ways. Unfortunately, I use them to host secondary DNS for all my domains, so this Network Solutions stupidity is hitting home for me right now. Then again, I had enough issues with NS back in the 1990s that as soon as that monopoly was broken I switched and never looked back. TBH I'm a little surprised that HE would use NS, but perhaps they did thinking that NS would provide proper enterprise-level support for a major backbone provider and not cut them off on a holiday with no recourse - behavior you might expect or fear from a cut-rate provider.
- toast0 2y ago
  Edit: since the domain is fixed, this isn't reproducable anymore. An interesting thing is that if you have a .net domain with he.net nameservers, the .net authoritative servers will give you full glue records with the A/AAAA for nsX.he.net. But if you ask for he.net, you get back NXDOMAIN. Example domain removed; no longer relevant. NetworkSolutions is a truly terrible registrar, and anyone who still has a domain there should use this occasion to switch to somewhere better, which is almost anywhere. IMHO, it may be worthwhile for a real business where their domain is important to move to an expensive corporate registrar like MarkMonitor or CSC, and looking into registry lock, etc.
- slyall 2y ago
  I guess this is one of the reasons AWS uses multiple domains in different TLDs for it's customer's name servers in route53[1] A provider going rouge or a domain expiring will probably still leave 3 perfectly working. [1] eg one of mine has awsdns-21.com, awsdns-50.co.uk, awsdns-11.net and awsdns-42.org
LinuxBender 2y ago
Looking to see what escalation numbers I may still have... [Edit]: Sent them what I have but it isn't much. Most human contacts have been replaced by menu's. I hope the last one I sent helps. It appears someone managed to get NetworkSolutions (now web.com) to put the domain he.net on clientHold for one Phishing complaint, unrelated to he.net meaning someone did a lookup on bgp.he.net and reported he.net instead of the domain they were looking up. If true it means web.com did not do any due diligence, investigation, etc... and just put a very popular domain with many shared services on hold. The domain he.net is used by MANY services and those services are likely going to start breaking. I have no idea what's going on over at web.com. As a side note, I recently transferred all my domains away from NetSol/web.com as they lost the ability to define custom name servers (glue in the root servers) which is bizarre to me, given many of the meme-style resellers of resellers can still do this. I had many other strange experiences with them, including having to teach their escalation contacts how to use dig outside of their own name servers.
- deleted 2y ago
  [deleted]
  - LinuxBender 2y ago
    I was with Network Solutions from 1997 to 2023. They were bought by web.com in 2018. They keep the name on the old domain and don't redirect to web.com because they have some non-API automation in place on the old domain.
    - qual 2y ago
      Wikipedia has it incorrect then, as they list it as "formerly web.com" ("Network Solutions, LLC, formerly Web.com is an American-based technology company"). Thanks for the clarification!
- fragmede 2y ago
  > As a side note, I recently transferred all my domains away from NetSol/web.com as they lost the ability to define custom name servers FFS is that what happened? They recently bought the registrar I use and a domain of mine with custom nameservers isn't working and I haven't had a chance to troubleshoot.
KennyBlanken 2y ago
https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en https://www.icann.org/resources/pages/epp-status-codes-2014-... > This status code tells your domain's registry to not activate your domain in the DNS and as a consequence, it will not resolve. It is an uncommon status that is usually enacted during legal disputes, non-payment, or when your domain is subject to deletion. > Often, this status indicates an issue with your domain that needs resolution. If so, you should contact your registrar to resolve the issue. If your domain does not have any issues, but you need it to resolve, you must first contact your registrar and request that they remove this status code.

speckx 2y ago

I can't believe that NetworkSolutions doesn't have a list of significant ISPs by domain name or something. I would proactively work with significant ISPs before suspending the entire domain and interrupting their services.

bifrost 2y ago
Its not really going to affect their services, maybe email in/out of it but thats about it. NetSol should've done better though, this makes them look like clowns.
- a-dub 2y ago
  > Its not really going to affect their services, maybe email in/out of it but thats about it idk. how do you know that their management capability can run sans dns?
  - bifrost 2y ago
    Because they run their own DNS servers and inside of their network they're going to be using their own infra...
    - a-dub 2y ago
      idk. aren't they tier 1? what if it breaks peering? is all their management capability strictly "inside" their network? what would happen to the rest of the internet if they dropped off?
KennyBlanken 2y ago
There's a reason people were practically partying in the streets when Network Solutions lost their monopoly. The fact that their netops / support staff have no way to escalate an issue like this is the bigger issue. IMHO the giant ISP shouldn't get a better response time or procedure; everyone should be able to get help in an emergency like this. Really, NANOG have themselves to blame. Half the reason NANOG was started was to help high level techies bypass low-level operations/support staff. Everyone should have worked together to make sure their own orgs could appropriately handle calls/issues like this. Instead, with the issue mostly fixed for them, they stopped caring, and all the peons and techies who don't work for Fortune 500 or famous tech companies are still fucked...

killerstorm 2y ago

[flagged]

Reviving1514 2y ago
Please elaborate, how would block chain solve this problem?
- bifrost 2y ago
  It would not.
  - deleted 2y ago
    [deleted]
- fragmede 2y ago
  If we treat Blockchain as a database with specific properties, and use that to back DNS, then we have a DNS system with those specific properties, instead of the system we currently have, which has different specific properties. Some of those properties are desirable, but as we see here, some of them are not.
- killerstorm 2y ago
  There's a number of DNS-like protocols on blockchain, e.g. ENS (Ethereum Name System - https://ens.domains/ https://ens.domains/), Handshake (https://handshake.org/ https://handshake.org/). They provide same functionality as DNS[sec]. You are not at whim of a corpo, nobody can suspend you, corrupting or hijacking records is pretty much impossible...
  - warkdarrior 2y ago
    That sounds like an excellent feature. Especially for phishing and other scammers who need bulletproof hosting.
    - killerstorm 2y ago

deleted 2y ago

[deleted]

codetrotter 2y ago

Status in whois response is “active” for me, and he.net resolves from DNS for me, so I guess it’s been fixed? But why did they list a @he.net email address as one of the emails to contact them on? Wasn’t the problem that he.net would not resolve because of that very client hold?

deleted 2y ago
[deleted]
KennyBlanken 2y ago
Confirmed at 3:30 CT
- codetrotter 2y ago
  I read a bit more in the linked email thread. Some other message said that caching would likely make it still resolve in most places and that checking the root servers would show the problem and that elsewhere it would show up around 48 hours after, when the caches expire. Still not sure if the status it shows in Whois means that it’s resolved or not.
- LinuxBender 2y ago
  Indeed was Domain Status: clientHold https://icann.org/epp#clientHold https://icann.org/epp#clientHold now Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited https://icann.org/epp#clientTransferProhibited Verified he.net is correctly listed in the root servers, though I did not check prior. dig +nocookie @i.gtld-servers.net he.net ;; AUTHORITY SECTION: he.net. 172800 IN NS ns1.he.net. he.net. 172800 IN NS ns2.he.net. he.net. 172800 IN NS ns3.he.net. he.net. 172800 IN NS ns4.he.net. he.net. 172800 IN NS ns5.he.net. ;; ADDITIONAL SECTION: ns1.he.net. 172800 IN AAAA 2001:470:100::2 ns1.he.net. 172800 IN A 216.218.130.2 ns2.he.net. 172800 IN AAAA 2001:470:200::2 ns2.he.net. 172800 IN A 216.218.131.2 ns3.he.net. 172800 IN AAAA 2001:470:300::2 ns3.he.net. 172800 IN A 216.218.132.2 ns4.he.net. 172800 IN AAAA 2001:470:400::2 ns4.he.net. 172800 IN A 216.66.1.2 ns5.he.net. 172800 IN AAAA 2001:470:500::2 ns5.he.net. 172800 IN A 216.66.80.18 which belong to Hurricane [1] [1] - https://bgp.he.net/AS6939#_whois https://bgp.he.net/AS6939#_whois

deleted 2y ago

[deleted]

basilgohar 2y ago

First off, Hurricane Electric has always been one of the "cool" companies on the Internet since the early days (of public Internet, at least). So, my goodwill with them tells me that Network Solutions, a company that most certainly has NOT been one of the cool companies, is at fault. I feel and know this to be the case now, as well. Having said that, I am also greatly pleased that the NANOG mailing list is public and plebes like us can see these kinds of discussions, which I feel are becoming harder to find as time goes on as services get consolidated into larger, government-friendly corporate private digital gardens.

deleted 2y ago
[deleted]

byyll 2y ago

Great news! It's a private company and they can do whatever they want! Just like HE likes to censor.

omoikane 2y ago

The hold appears to have been removed as of few minutes ago: https://mailman.nanog.org/pipermail/nanog/2024-July/225903.html https://mailman.nanog.org/pipermail/nanog/2024-July/225903.h... I didn't know HE.net was using Network Solutions, maybe they will switch to someone else soon. My personal experience with Network Solutions wasn't so great (lots of trouble and human intervention needed to transfer a domain out of their service).