12 ms·
.IO domain name reliability issues and how we’re working around them
- feikal 9y agoi always wondered why hipsters use a strange "indian ocean" domain for serious stuff.
- advisedwang 9y agoI wonder if this is a factor in why large tech organizations like Amazon, Facebook, Google, Microsoft etc have been so keen to register their own TLDs. Now they only depend on root NS and their own infrastructure, and not _any_ ccTLD provider.
- gist 9y agoIf that were the case then Amazon and practically everyone else wouldn't be running under .com which of course is managed by Verisign. Will also note that Amazon is a registrar but relies (as others do) on markmonitor.com to be the registrar for their own domain .com name. Ditto for google which is also a registrar. So they could be the registrar for their own .com domain but choose not to do so.
- CydeWeys 9y agoDomain names are amazingly sticky. You'll likely start seeing increased news of companies using their closed TLDs, but don't expect too many existing sites moving over.
- CydeWeys 9y agoYes, this is one of the major reasons that we registered our own TLDs. It's definitely not a secret. Source: I'm the tech lead of Google Registry.
- mmerlin 9y agoAny hint of a timeline for .app availability? https://www.registry.google/about/domains.html https://www.registry.google/about/domains.html
- CydeWeys 9y agoStay tuned.
- jameskegel 9y ago...wink one time if measured in weeks, wink two times if measured in months
- CydeWeys 9y agoWell it's definitely not weeks. The rollout process for a TLD takes several months, most parts of which are unavoidably public.
- Veelox 9y agoYou really expect to hear a product update via HN comment?
- CydeWeys 9y agoWe do actually find these kinds of comments helpful, because it lets us know which of our TLDs are in highest demand.
- JdeBP 9y agoGiven the likes of https://news.ycombinator.com/item?id=15353027 https://news.ycombinator.com/item?id=15353027 and https://news.ycombinator.com/item?id=15269832 https://news.ycombinator.com/item?id=15269832 , it is not beyond the bounds of possibility.
- dx034 9y agoMay I ask why Google isn't using it's own TLD yet? I just tried combinations such as mail.google, maps.google or news.google and would've expect that they at least redirect to the equivalent.
- joering2 9y agoWith so many issues related to non .com TLD, I was in deep shock to learn Google registered abc.xyz
- thisisit 9y ago> Back when we started in 2014 we decided that .io was great from a branding perspective. Stream is a technical product and our audience is mainly technical, so .io seemed like a great match. Using the same domain for the APIs was more of a consequence than a thoughtful decision. Quite a lot of tech companies are using .io nowadays. If .io reliability is an issue what is the next best alternative for tech domain names?
- deleted 9y ago[deleted]
- untog 9y agoJust use .com for anything critical (i.e. that your customers rely on). Keep a .io domain or whatever you want for marketing purposes if it's absolutely necessary, just realise your splash page is at risk of failure some times. Whether it's worth the trade-off is likely up to the individual company.
- yeukhon 9y agoMarketing is the wrong drive. .com is already over populated meaning you can't get the domain you really want. I get a couple domains in .io so I am happy.
- qq66 9y agoYou can use stream.io for marketing, email, etc. and use something like stream-dot-io.com or 2556369863227.com for production APIs.
- wonderwonder 9y agoThis is the correct answer
- fancyfacebook 9y ago"Correct" but feels really gross
- foepys 9y agoI never understood why ccTLDs of some third-world countries became popular for hosting production code. There is a real risk of somebody stealing your domain due to vulnerable infrastructure and incompetence at the organizations that sometimes employ less than 10 people. The notable takeover of the .io ccTLD by Matthew Bryant [1] should have been a wakeup call for everybody. 1: https://thehackerblog.com/the-io-error-taking-control-of-all-io-domains-with-a-targeted-registration/index.html https://thehackerblog.com/the-io-error-taking-control-of-all...
- nkrisc 9y ago> I never understood why ccTLDs of some third-world countries became popular Simple vanity.
- tschellenbach 9y agoI think it's also about awareness. Most people don't know that IO or other smaller TLDs are unreliable.
- CamperBob2 9y agoThey aren't unreliable. This article blows up a single minor incident into a controversy of epic proportions.
- sokoloff 9y agoThe incident was far from minor IMO. I think .io has had 6+ hours of downtime (full or partial/non-trivial) this year, long enough to push the annual availability down to 99.93% from what I recall from memory of reading post-mortems. That was enough to spur us to begin planning to move production traffic off our .io domain as well.
- dx034 9y agoDid any of the large TLDs (.com/.net/.de/.fr/.co.uk) have an outage like that in the last decade? Not being available for several hours can severely damage businesses. Startups that haven't built trust yet as well as some large companies. Imagine a bank's website not being available for several hours. It doesn't matter if it's a DNS issue, people would start panicking very quickly.
- bryanlarsen 9y agoBy that argument you should also stop using: .org, .ngo, .lgbt, .asia, .aero, .info, .mobi, .pro, , .MN (Mongolia), .AG (Antigua and Barbuda), .BM (Bermuda), .BZ (Belize), .GI (Gibraltar), .IN (India), .ME (Montenegro), .SC (the Seychelles), and .VC (St. Vincent and the Grenadines), .SG (Singapore) and .HN (Honduras) and more All are provided or supported by Afilias, the company running .IO. https://en.wikipedia.org/wiki/Afilias https://en.wikipedia.org/wiki/Afilias
- Multicomp 9y agoFWIW a glance at that wikipedia page does not show any references to the .IO TLD. According to https://en.wikipedia.org/wiki/.io https://en.wikipedia.org/wiki/.io .IO TLD is run by Internet Computer Bureau - are they the same/related companies?
- bryanlarsen 9y agohttps://www.internetnews.me/2017/04/12/afilias-wins-contract-run-backend-registry-services-io-ac-sh/ https://www.internetnews.me/2017/04/12/afilias-wins-contract...
- deleted 9y ago[deleted]
- andyhmltn 9y agoIt's hardly surprising sadly. Looking back through my submission history there's two instances of incompetence by the .IO handlers: 1) Storing passwords in plain text: Although the post is now down, it pointed to the fact that Nic.IO will email you your password in plain text. Something which everyone that's spent more than a few weeks coding should know not to do. 2) Another pretty major outage back in 2013 that rendered two of my sites offline. I personally stopped using them after my last IO domain lapsed but this should serve as a stark warning to anyone thinking they can pick up a cute IO domain
- quocble 9y agoSaid by the company that uses .io
- andyhmltn 9y agoI mean if you read the article the reason they say not to use it is specifically because they used it themselves and were burnt
- deleted 9y ago[deleted]
- MistahKoala 9y agoThere is also a good, ethical reason to stop using .io domains completely, as the territory's population had no say in the governance of the TLD and are deprived of any revenue from it.
- jandrese 9y agoIs it expected that the population of a country would benefit directly from TLD sales? I have not seen a dime of the proceeds from .us. I guess the money probably goes into the treasury, but it can't be more than a drop in the bucket. I still think TLDs in general were a mistake. It always felt life a leftover from the old Usenet dominated Internet. The Internet quickly outgrew the categorization system and we ended up where we are today with most everything being shoved in .com because it was the least strict.
- 45h34jh53k4j 9y agoIts far more than this, the original inhabitants of .io were forcibly removed from these islands by the UK in the 60s. The whole situation with a UK company running this TLD is just the icing on the cake. gigaom.com/2014/06/30/the-dark-side-of-io-how-the-u-k-is-making-web-domain-profits-from-a-shady-cold-war-land-deal/
- pmyteh 9y agoThe population of the British Indian Ocean Territory were all deported, to clear the island of Diego Garcia for a large US airbase. It's one of the grubbiest recent bits of British imperial manoevering.
- eponeponepon 9y agoGrubby is an understatement. We "persuaded" people off the island by (for instance) not letting them back in when they returned from holidays, rationing their food and drink, and literally gassing their dogs in front of them. Edit to add: just to be clear, the UK had bought these people's homeland something like five years previously. Bought.
- hodgesrm 9y agoDumb question, but why not host DNS name serving somewhere other that NIC.io so it does not go down? As I recall you can set the name server to live somewhere else, though I don't have the account pages open. The dnscheck.pingdom.com page shows that .io domains commonly route to different name servers. Just try stream.io and slither.io and you should see different authoritative name servers. (I picked those randomly for the test.)
- mikepavone 9y agoDNS resolution is hierarchal. You start at the root DNS servers for a TLD to find the authoritative servers for a given domain in that TLD. You can host your authoritative servers wherever you want, but you have no control over the root servers and it was the latter that had problems.
- hodgesrm 9y agoThanks, that clears things up.
- ryandrake 9y agoIf these are just API endpoints (not typed in by users, and thus, no Marketing value to the name) why not just register <random hash>.com?
- tschellenbach 9y agoThat's what we do :) It's ok for the marketing page to go down every now and then. Not so nice for the API to become unavailable. (200 million end users)
- mschuster91 9y ago> In the first case, we would need to keep hundreds of DNS records in sync and double our SSL certificates; The first thing is admittedly a PITA, but SSL certificates should not be a problem. Either you use LetsEncrypt which automates the pain away anyway or you do the sane thing and buy a wildcard cert - this has the added advantage that no one can run a service discovery by simply grepping a CT log. Yes, I know, security by obscurity, but scriptkiddies will go for the low hanging fruit first, and having your domains show up in CT logs is ultra low hanging fruit. > secondly we would need to only change our infrastructure to not use any Route53 specific feature You should not be locked in to Amazon (or for that matter any Cloud provider) anyway, given how easy it is to get banned from them (hint: it's enough if your Google account manages also the Youtube channel and videos on it get striked too often). > Using a widely used TLD like .com/.net/.org is the best and easiest way to ensure reliability. Another caveat right here: .com and .net are operated by Verisign, while .org is operated by PIR. You should always take care to choose a different operator for the backup TLD!
- marknadal 9y agoWe have a method for keeping thousands of DNS records in sync per second. Would this by chance be useful? Who should I get in contact with to help?
- dan15 9y ago> Either you use LetsEncrypt which automates the pain away anyway or you do the sane thing and buy a wildcard cert For what it's worth, Let's Encrypt plans to offer wildcard certs starting in January 2018.
- tbarbugli 9y agoauthor here: About your second point. We spend quite some money on AWS every month. As long as we keep paying our bills, I don't see why they would want to cut us loose.
- mschuster91 9y ago> As long as we keep paying our bills, I don't see why they would want to cut us loose. I can imagine a competitor trying to boot you off by bombarding their abuse team with bogus complaints and triggering automated actions. It's been done in the past, the problem is that all major companies (not just hosting, but ANY large company!) try to save on actual customer support and even more on stuff they can't bill to the customer, like a properly staffed abuse team.
- 45h34jh53k4j 9y ago.io represents the suppression of the Chagossian people. https://en.wikipedia.org/wiki/Chagossians https://en.wikipedia.org/wiki/Chagossians English imperialism at its best. Do not use this unethical TLD: https://gigaom.com/2014/06/30/the-dark-side-of-io-how-the-u-k-is-making-web-domain-profits-from-a-shady-cold-war-land-deal/ https://gigaom.com/2014/06/30/the-dark-side-of-io-how-the-u-...
- porfirium 9y agoMight makes right.
- gmiller123456 9y agoBy those standards, can you provide a TLD that is not the result of some atrocity or another?
- 45h34jh53k4j 9y ago.cx? (Christmas Island), which is a Australian territory? Maybe your right. I think .io is especially egregious because of history of Chagossian people.
- briandear 9y agoWhere they planning on setting up a TLD? It seems like you have to have an almanac before you do anything these days, lest you offend someone.
- timc3 9y agoFor anyone recommending .com one of the simple factors that it isn't often a good choice is people sitting on domain names for profit or other reasons. So either you have to think up a completely ridiculous name for your new service/product/company, pay stupid amounts or use something other than .com
- wonderwonder 9y agouse .io or similar for any customer facing websites but a .com for any mission critical api type calls. Customer never even has to know about the existence of myUglyCompanyNameApiHandler.com but all api calls are handled by it.
- OhHeyItsE 9y agoIt's a good idea, but if your product is a website, cookies and authentication in general are going to be a real challenge. Obviously you won't be able to share cookies between the site and the api. But you have the additional challenge of having your api domain treated as 3rd party and are thus subject to all the security measures and sandboxing that come with that. And what if your product is just an api for use by developers? Now the api is part of your brand. It's just a bad situation all around.
- dan15 9y ago> Obviously you won't be able to share cookies between the site and the api. I don't know of any API that requires cookies from the site. Usually you use an access token or something similar for an API. > But you have the additional challenge of having your api domain treated as 3rd party and are thus subject to all the security measures and sandboxing that come with that. CORS solves most of the pain points. You just need to ensure the API is serving the correct CORS headers.
- wonderwonder 9y agosure its not optimal but the lack of available .com domain names forces you into a corner at least in the particular situation referenced by OP. The other poster is correct though in that CORS will handle alot of your issues. You are absolutely right in that it is definitely going to increase the effort involved in rolling out a site/service. I don't know of any pretty solution to the issue.
- rdiddly 9y agoTL;DR - company follows trend of misusing top-level domain of the British Indian Ocean Territories, finds it unreliable and not-so-great, starts the wheels in motion to switch away from it. Addendum: Mind you I applaud their coming forward to report this. Hopefully people get the message. It's not the first time fashion got the better of a large number of people.
- yeukhon 9y agoSorry, but the title is click bait. Can people just write "Our IO domain Resolution Failed, A Lesson Learned"? In the "What Really Went Wrong" section, the author wrote: > it does not take a lot of research to find out that the .io TLD team made several mistakes and went on to cite two incidents, which are not exclusive to .IO domain. So let's not blame ".io domain team" and only happen to .IO domain, try to convince me (at least the way I read it based on the title) that using .io domain is a bad idea. I should stop using .com then. For me, I was looking for a "so we are going to switch away from .io domain in the next year or so" at the end of the post, because .io is not good according to the author. The plan instead is just add a backup domain, so it looks like the author is eating his/her own words, even though the critical stuff are going to run over .com. So let's just switch everything over, what's the big deal for your user-facing website not over .com? Do people really think a big deal now about .io vs .com when you have established a business? Anyway, I do appreciate when a postmortem is available because I can learn something new, but I do hope we write postmortem with an objective tone. So no, if there is a downvote button on HN for the submission, I would downvote.
- manigandham 9y ago.com/net/org are also faster, especially internationally, because of more infrastructure. Algolia found the same during their testing [1]. Use the main TLDs for serving traffic and .io/* for corporate/marketing sites or webapps if the occasional outage isn't a major problem. 1. https://stories.algolia.com/algolia-s-fury-road-to-a-worldwide-api-c1536c46f3a5 https://stories.algolia.com/algolia-s-fury-road-to-a-worldwi...
- dan15 9y agoSome ccTLDs have good infra though. Many of the smaller ones (particularly ones ran by CoCCA, such as .cx) use Dyntld or Packet Clearing House which have pretty good servers and connectivity.
- quirkot 9y agoIt's not my area, but it seems like if your application is deeply dependent on an API it might be a good idea to directly reference the IP address? Over 1000s of requests, wouldn't that save time?
- mikeash 9y agoIt would be slightly faster. However, if you ever had to change your server's IP address (for example, because you had to move to a new hosting provider) you'd be utterly screwed.
- metalliqaz 9y agoThat scheme assumes that the service is hosted at a stable IP address, which in almost all cases is not a valid assumption.
- elsombrero 9y agoNot my area either but I think dns queries are cached locally for a certain amount of time so dns lookup for 1000s of requests shouldn't give too much overhead Edit: yeah and also what the other guys said about not relying on the ip being the same forever
- gator-io 9y agoRemember this: https://www.icann.org/news/announcement-2017-09-27-en https://www.icann.org/news/announcement-2017-09-27-en I think .io was part of the issue for the delay. After the last outage, we at gator.io took it as a 'all hands on deck' firedrill to get our api off .io. The problem is that many of our customers have scripts on their sites with the old .io endpoint. Migrating them is very difficult.
- ca98am79 9y ago"Due to its decentralized nature..." should read "Due to its CENTRALIZED nature..." If DNS were decentralized, like for example Bitcoin, so many problems like this would never happen.
- numbsafari 9y agoTrue... until "devops199" randomly gives you 150 million reasons otherwise.
- tbarbugli 9y agoauthor here: this indeed incorrect, I will change that into "centralized and distributed" thank you for pointing that out
- deleted 9y ago[deleted]
- ca98am79 9y ago.io is managed by Afilias. They took over and migrated the registry to their system in June. Afilias manages many other TLDs including: .adult .ag .archi .bet .bio .black .blue .bz .global .green .in .info .kim .lgbt .ltda .me .mn .mobi .ngo/.ong .org .pet .pink .poker .porn .promo .red .sc .shiksha .vc .vegas .vote .voto They are a very reliable registry operator. This was a bad screw up, and I guess it had something to do with the migration.
- feelin_googley 9y ago.io domains are perhaps fashionable, but they are expensive. For $0.88-$0.99/yr one can have a domain in the same registry as the Alphabet Inc. website's domainname. For that price, it would not be a clever name, but it could be an easy-to-memorize 6-digit number. What if it is only used for an API endpoint? A higher level of "reliability" IMHO could be achieved by use and publication of a stable IP address, perhaps anycasted if one can afford it. At least it could be a backup for emergencies, such as DNS failures. Consider that DNS itself e.g., disseminiation of root.zone, does not rely on DNS. The IP address for ftp.internic.net is well known and rarely changes. As I recall, when it does, they notify the public in advance. Some years ago if I am not mistaken, there was a change from 208.77.188.26 to 192.0.32.9. Another example is third party DNS providers. They too publish stable IP addresses. Sometimes users might even memorize them, or store lists of these addresses e.g. included within installed software. As a user, I hold no bias against any company that publishes its stable IP addresses. In fact, on the issue of reliablity I would hold them in higher regard than those who rely 100% on DNS and third parties associated with DNS service. DNS is reasonably reliable, but IMO not more so than a stable IP address.
- lcurole 9y ago"In July 2017 a security engineer from Google was able to buy the domain of one of the authoritative nameservers (ns-a1.io) and gain control of every .io domain." Ouch.
- stefanukchagos 9y agoHello, I volunteer with UK Chagos Support Association, a voluntary Chagossian support group that Plan.io and other .io firms and users have donated to since this issue was raised in a tech magazine a year or two ago. Someone Twitter tagged us and linked to this discussion and it's great to see so much support for the community. Couple of links on the background below. http://archive.chagossupport.org.uk/index.php/category/io/ http://archive.chagossupport.org.uk/index.php/category/io/ https://gigaom.com/2014/07/02/seats-io-donates-money-to-chagos-islands-charity-over-io-domain-use/ https://gigaom.com/2014/07/02/seats-io-donates-money-to-chag... A couple of .io start ups set up a site to encourage others to support us and other Chagos-related support groups which allowed us to a lot more of our work - campaigning, supporting community projects and issuing crisis grants - over the past few years. Called 'The Dark Side of .Io,' it's actually offline now but the owner has assured me it'll be back on shortly. I've no expertise to comment on technical issues. On the moral side, I've not met anyone in the community who has a serious issue with firms using the .io domain name - most are impressed that so many firms have chose to back the Chagossian community out of their own pocket. As someone mentioned below, as recently as one year ago the UK government refused to allow Chagossians to return to the Chagos Islands. The community will not see any money from the renewal of the lease on their homeland for use as a military base. These are the main issues, and while the .io domain name is somewhat symbolic of the wider exploitation of the Chagossian people - others profiting from their homeland with the community itself seeing no benefit - there is at least a positive side to this, which is harder to find in the government's decision to continue the exile. If anyone is interested in finding out more please see our website (not .io - but that's out of cost and ignorance factors more than ethics) www.chagossupport.org.uk We're actually planning a bit of a revamp of the site shortly and I'd be remiss if when posting here I didn't ask for anyone interested in getting involved in that - contact@chagossupport.org.uk
- zx2c4 9y agoWireGuard similarly moved from the .io to wireguard.com in July: https://lists.zx2c4.com/pipermail/wireguard/2017-July/001569.html https://lists.zx2c4.com/pipermail/wireguard/2017-July/001569... Less trendy, but otherwise, zero regrets.
- hoodoof 9y agoNice, if you have the .com
- elliotanderson 9y agoWe migrated from serving production JS assets off our .io domain to .com after we found a number of corporates/schools in Ireland blocking the domain at the TLD level. Since then, the number of inexplicable error reports has dropped dramatically.
- andyonthewings 9y agoI guess that's why the domain name infrastructure should be decentralized. Though I'm not familiar with the tech involved, does blockchain based things, like namecoin, solve this problem?
- henvic 9y agoCross-posting a message I posted about one year ago because I am still mad about what happened 7 years ago: "In 2011 I paid for and registered o1.io (I really wanted 01.io, but back then it was not available to register domains containing only numbers on .io). The .io NIC web interface is really horrible and I ended up in an inconsistent state after making the "horrible" mistake of clicking the back button. Even though my Paypal account got charged, I received an email confirming I was the owner of the domain, and so on, I couldn't access my account. Next step I took was contacting them. I got ignored and after 7 days later they made a transaction reversal on Paypal and I never heard a word back from them, even though I have tried to contact. Some time later they made the o1.io domain a reserved one and so if I wanted it now I would have to pay thousands. What a shame." https://news.ycombinator.com/item?id=12980040 https://news.ycombinator.com/item?id=12980040