5 ms·

IO domain DNS failure

35 points by yoshiokatsuneo 10y ago

For 3 or 4 hours, IO domain DNS lookup often fails. Now, only 2 out of 7 IO domain nameservers respond, and other 5 DNS nameservers are dead.

a10c 10y ago
I have been seeing the same failures for many IO TLDs https://www.whatsmydns.net/#A/gitlab.io https://www.whatsmydns.net/#A/gitlab.io Example: https://i.imgur.com/SU4BAHK.png https://i.imgur.com/SU4BAHK.png I have noticed the failures occurring moreso in Asia and Russia than in other regions.
yoshiokatsuneo 10y ago
There are 7 IO nameservers (ns-d1.io, ns-l1.io ,ns-y1.io, ns-a1.io, ns-a2.io, ns-a3.io). Only ns-a4.io and ns-a3.io respond.
asher_ 10y ago
We are experiencing the same from Vietnam
mic159 10y ago
We are also having issues resolving some .io domains from AWS Sydney (ap-southeast-2)
tkazec 10y ago
We've been seeing this for exactly five hours now, all across East Asia. Edit: Looks like .io is managed in London, and their customer service off hours match the downtime.
user9999 10y ago
Having the same issue in Singapore. got nothing to do, helpless.
yoshiokatsuneo 10y ago
So, "ns-a4.io" is the culprit! On ns-a4.io, xxx.io can be resolved, but yyy.xxx.io cannot be resolved(delegated). Ref: http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39 http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39
yjchang 10y ago
Same here in South Korea. Still not fixed.
xiaodown 10y ago
Getting significantly worse now.
lifeeth_ 10y ago
.sh and .ac are also run by the same folks - looks like there are issues with those tlds as well.
alwaysanon 10y ago
Two of their servers were sending out NXDOMAINs which are then cached by downstream forwarders. It seems they have turned those off now.
yoshiokatsuneo 10y ago
It looks, the the problematic nameserver "ns-a4.io" is down. So, the problem should be disappeared... At the same time, it looks, there is only one "ns-a3.io" nameserver working for IO domain, at now...
- xiaodown 10y ago
  Yep, rather than no such domain or a bad delegation, the "bad" servers seem to just be down now.
JonAtkinson 10y ago
I recieved this reply from nic.io support a few minutes ago: http://imgur.com/a/wV2Kk http://imgur.com/a/wV2Kk
matt_oriordan 10y ago
This is still affecting us. Our customers on Heroku seem to be the worst affected as Heroku is unable to resolve the DNS
xiaodown 10y ago
AWS response: Hello, Some customers have reported intermittent resolution issues with .io domains. We can confirm that Route 53 DNS services are operating normally at this time and these issues seem to be related to the .io top-level domain provider. We've investigated the issue and are primarily seeing incorrect responses from two of the .io nameservers: ns-a4.io and ns-a2.io. These nameservers are returning NXDOMAIN intermittently for domains that do exist. As a result, once a resolver receives the erroneous response, it will cache the non-existence for the negative TTL, which for .io is set to 3600 seconds (1 hour). One suggestion we have is to increase the TTLs for your domains in Route 53 so that the resolvers cache DNS answers for a longer duration. Best regards, (Customer support rep name redacted) Amazon Web Services We value your feedback. Please rate my response using the link below.
jansh 10y ago
For Planio (https://plan.io/ https://plan.io/), all accounts are available via yourname.planio.net as an alternative while this outage lasts.
matt_oriordan 10y ago
For those interested, we wrote a blog post about single points of failure with DNS last Friday when the Dyn attack happened. Our strategy ensures our customers are mostly unaffected by issues like this. See https://blog.ably.io/routing-around-single-point-of-failure-dns-issues-7c20a8757903 https://blog.ably.io/routing-around-single-point-of-failure-...

For 3 or 4 hours, IO domain DNS lookup often fails. Now, only 2 out of 7 IO domain nameservers respond, and other 5 DNS nameservers are dead.

a10c 10y ago

I have been seeing the same failures for many IO TLDs https://www.whatsmydns.net/#A/gitlab.io https://www.whatsmydns.net/#A/gitlab.io Example: https://i.imgur.com/SU4BAHK.png https://i.imgur.com/SU4BAHK.png I have noticed the failures occurring moreso in Asia and Russia than in other regions.

yoshiokatsuneo 10y ago

There are 7 IO nameservers (ns-d1.io, ns-l1.io ,ns-y1.io, ns-a1.io, ns-a2.io, ns-a3.io). Only ns-a4.io and ns-a3.io respond.

asher_ 10y ago

We are experiencing the same from Vietnam

mic159 10y ago

We are also having issues resolving some .io domains from AWS Sydney (ap-southeast-2)

tkazec 10y ago

We've been seeing this for exactly five hours now, all across East Asia. Edit: Looks like .io is managed in London, and their customer service off hours match the downtime.

user9999 10y ago

Having the same issue in Singapore. got nothing to do, helpless.

yoshiokatsuneo 10y ago

So, "ns-a4.io" is the culprit! On ns-a4.io, xxx.io can be resolved, but yyy.xxx.io cannot be resolved(delegated). Ref: http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39 http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39

yjchang 10y ago

Same here in South Korea. Still not fixed.

xiaodown 10y ago

Getting significantly worse now.

lifeeth_ 10y ago

.sh and .ac are also run by the same folks - looks like there are issues with those tlds as well.

alwaysanon 10y ago

Two of their servers were sending out NXDOMAINs which are then cached by downstream forwarders. It seems they have turned those off now.

yoshiokatsuneo 10y ago

It looks, the the problematic nameserver "ns-a4.io" is down. So, the problem should be disappeared... At the same time, it looks, there is only one "ns-a3.io" nameserver working for IO domain, at now...

xiaodown 10y ago
Yep, rather than no such domain or a bad delegation, the "bad" servers seem to just be down now.

JonAtkinson 10y ago

I recieved this reply from nic.io support a few minutes ago: http://imgur.com/a/wV2Kk http://imgur.com/a/wV2Kk

matt_oriordan 10y ago

This is still affecting us. Our customers on Heroku seem to be the worst affected as Heroku is unable to resolve the DNS

xiaodown 10y ago

AWS response: Hello, Some customers have reported intermittent resolution issues with .io domains. We can confirm that Route 53 DNS services are operating normally at this time and these issues seem to be related to the .io top-level domain provider. We've investigated the issue and are primarily seeing incorrect responses from two of the .io nameservers: ns-a4.io and ns-a2.io. These nameservers are returning NXDOMAIN intermittently for domains that do exist. As a result, once a resolver receives the erroneous response, it will cache the non-existence for the negative TTL, which for .io is set to 3600 seconds (1 hour). One suggestion we have is to increase the TTLs for your domains in Route 53 so that the resolvers cache DNS answers for a longer duration. Best regards, (Customer support rep name redacted) Amazon Web Services We value your feedback. Please rate my response using the link below.

jansh 10y ago

For Planio (https://plan.io/ https://plan.io/), all accounts are available via yourname.planio.net as an alternative while this outage lasts.

matt_oriordan 10y ago

For those interested, we wrote a blog post about single points of failure with DNS last Friday when the Dyn attack happened. Our strategy ensures our customers are mostly unaffected by issues like this. See https://blog.ably.io/routing-around-single-point-of-failure-dns-issues-7c20a8757903 https://blog.ably.io/routing-around-single-point-of-failure-...