7 ms·
GoDaddy outage caused by corrupted router tables
- xtdx 14y agoNot a good week for claiming credit...
- hbz 14y ago"The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack." So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? The router tables just corrupted themselves?
- sendos 14y ago> So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? Yeah, with a name/handle like Anonymous0wn3r they sound very trustworthy. If they claimed they had a hand in something, it must be true.
- mbell 14y agoI don't trust Anonymous0wn3r but I distrust GoDaddy just the same. Of course those aren't the only two answers, it could have been external and not Anonymous0wn3r as well.
- andreasvc 14y agoBut didn't he announce the attack before-hand?
- 46Bit 14y agoGot a link on that?
- andreasvc 14y agohttp://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/?utm_source=dlvr.it&utm_medium=twitter http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-m... Not sure if those tweets were actually from before it happened.
- sendos 14y agoI searched for those tweets mentioned in the Techcrunch article (first from GoDaddy and first from AnonymousOwn3r on this topic) and the time stamps are: * GoDaddy: 10:35 AM - 10 Sep 12 * AnonymousOwn3r: 11:57 AM - 10 Sep 12 So, AnonymousOwn3r does not seem to have announced the attack before it happened. Links to tweets: * https://twitter.com/GoDaddy/status/245213898683318272 https://twitter.com/GoDaddy/status/245213898683318272 * https://twitter.com/AnonymousOwn3r/status/245234582205652992 https://twitter.com/AnonymousOwn3r/status/245234582205652992
- ceejayoz 14y ago> So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? Gee, why would an anonymous person on the internet lie?
- TomGullen 14y agoI don't know much about hardware at all, but aren't routers fairly simple, time tested pieces of hardware? Can they really corrupt en-masse in this way?
- sharth 14y agoRouters can get quite complicated. At a first level, routers will cache information about what machines are on which port of the router. This enables them to not forward a packet to every port on the router (reducing network load). This is normally done using MAC Addresses. On more expensive routers, the router can understand IGMP, or multicast and route based on multicast joins. This enables an optimization on simple broadcasts because not every machine on the network needs to see these packets, but multiple machines might want to.
- nettdata 14y agoNever mind when you start adding fault-tolerance and failover setups/configs to the mix. It gets really complex really fast, and even the slightest lapse in attention to extreme detail can be disastrous and a nightmare to troubleshoot. For instance, I once had a router reboot for some unknown reason, and the firmware/config wasn't properly flashed, so it reverted to the last point release of the firmware. That was enough to cause the failover heartbeat to be constantly triggered, and the master/slave routers just kept failing over to each other and corrupted all the ARP caches. Makes for a lot of fun when things work and then stop working in an inconsistent manner.
- tlb 14y agoThey're not that simple due to aggressive speed optimizations like caching frequently used routing information in hardware registers. If the cache gets corrupted due to electronic bogon flux, bad things happen. These events are rare (like once every billion operating hours) so you can imagine it's hard for the router vendors to find & fix them all. It is a little sad that they didn't try rebooting the routers for so many hours.
- jey 14y ago
- oomkiller 14y agoThis doesn't do anything to explain why it was out for so long. I guess I should expect this type of thing from GoDaddy though, they are mainly a consumer company.
- mbell 14y ago"We have determined the service outage was due to a series of internal network events that corrupted router data tables." I'm witholding any judgement on internal vs external involvement till this series of events is defined. (doubt it ever will be)
- staunch 14y agoBased on a long history of working in datacenters I'd bet someone misconfigured something and later claimed it was "corrupted" to save their ass - happens all the time. It's just so simple to make very confusing and damaging mistakes in a complicated network. I wouldn't be surprised to hear that GoDaddy's corporate culture wouldn't respond well to someone admitting to a mistake this damaging.
- dclowd9901 14y agoEx-GoDaddy employee here: Everyone there is on pins and needles at this point. Since Silverlake's investment in the company, many hatchets have dropped on jobs, and it's really the only decent tech firm in Phoenix to work at. My guess is that there is some hiney covering going on with this explanation, and the interim CEO has little cause to care too much about responsibility, since he'll likely be out before year's end anyway. I really feel for the folks who work there. Many, many talented people who don't have an inch to make a mistake. When I was last there, they had just released an internal communication about the new company motto: "It won't fail because of me." Horrible, horrible, backward-ass culture.
- citricsquid 14y agocan you clarify what you mean by: > it's really the only decent tech firm in Phoenix to work at. and then: > Horrible, horrible, backward-ass culture. what part of the company is good if not the culture?
- overworkedasian 14y agoI wasnt really paying attention to the outage, but if it was indeed a routing issue, then you shouldnt have been able to reach any godaddy ip address. ICMP/traceroutes would have failed and showed the error.
- nettdata 14y agoWhy would you say that? Large networks have a ton of routers in them, and even a lot of switches provide routing functionality. I don't know the details of the environment, but even in smaller systems I've worked on there is a fair bit of hardware separation between various network segments. Complete failure on one part would not affect the others. For that matter, even a slight corruption in some ARP caches, or stale internal tables, etc., could cause the problems they had... it's not just a complete failure that could cause problems. And "routing" is such a generic term, when it could really be any number of feature sets that failed; load balancing, source routing configs, etc.
- andreasvc 14y agoYou're forgetting internal routing; e.g. unreachable database server.
- noselasd 14y agoThey havn't said where the router issue was. I'd be surprised if their public facing DNS servers are actually storing the data as well - more likely the actual records are in a SQL, LDAP or similar backend on their internal network - which will have a huge internal infrastructure, including many, many routers.
- kijeda 14y agoThe DNS is designed to provide resiliency to these kinds of problems by providing the ability to list multiple NS records located in different networks. It is standard practice for top-level domain operators and other high-activity domains to place their name servers in different networks to guard against these kinds of issues. When companies put all their name servers in the same network, they are removing the diversity benefit and create a single point of failure. Domain operators should take this as a cautionary tale that they shouldn't have all their eggs in one basket and make sure a single network failure couldn't take all their name servers offline.
- andreasvc 14y agoThere are many other single points of failure besides network failure, such as pushing the wrong configuration. In fact it seems to me that it would be rather rare for a multi-homed datacenter to have a network failure.
- kijeda 14y agoIf you have a routing issue, whether it is due to "corruption" or misconfiguration, having some of your name servers on an entirely different network (i.e. a different AS) with a different routing policy is not going to be affected.
- ww520 14y agoWhat is their 99.9% SLA liability going to be?
- xyzzyb 14y agoZero. .001 of 1 year is just under 9 hours of downtime. http://www.wolframalpha.com/input/?i=.001+*+1+year http://www.wolframalpha.com/input/?i=.001+*+1+year
- RKearney 14y agoNothing, since 99.9% allows them nearly 9 hours of downtime a year.
- ww520 14y agoDidn't they have more than 9 hours of outage?
- druiid 14y agoI find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their IP addresses could be pinged which further goes to disprove a routing issue. More than likely high traffic crashed one or more routers (THIS I have seen happen) and the live/saved configs didn't match. I'd put more money on something like this happening if it was router related.
- nettdata 14y agoUnless you have intimate knowledge of their network topology, and know the specifics of where those pinged IP's live in that topology, and what routes were used to provide DNS results, you can't say that it wasn't a routing issue. "Routing" is a rather generic term when it comes to large networks, and everything from border routers, firewalls, load balancers, and switches actually perform routing. Especially (as I've mentioned in another post) when you add fault tolerance / failover configurations to the mix. Routing failure doesn't have to be an all-or-nothing thing. There are a number of ways in which I can see ICMP echo packets working but other traffic not, especially when you include complexities of source routing, load balancing, failover, etc. Even something as "simple" as a poisoned ARP cache in a single box could screw up the entire internal network and cause the problems they've had, and still be considered a "routing issue". $0.02
- druiid 14y agoNone of that is necessarily incorrect... but per their news release 'corrupted router data tables' (their words) were the issue. I can't read too much into that, but that still doesn't change the fact that DNS wasn't resolving for a while after they made their Verisign change (for clients), yet their website was resolved with this change. You are correct that I don't know the details of their internal network and I never said otherwise, just that the chain of events and their claims don't necessarily match up!
- highfreq 14y agoHow can they claim 99.999% uptime, when they just had several hours of service outage? I'm not sure how long they've been providing DNS hosting, but by the most generous assumption this would be the entire 15 years of their existence. 99.999% allows them about 1.3 hours of outage in 15 years.
- mikedouglas 14y agoI think what they're trying to say is that 99% of the time, they have 99.999% uptime.
- JoeAltmaier 14y agoSilly: 99% of the time they have 100% uptime.
- dllthomas 14y ago60% of the time it works every time?
- JoeAltmaier 14y agoWas responding to the silly parent post, claiming similar nonsense.
- dllthomas 14y agoYes, wasn't criticizing, just enjoying.
- lamebrain 14y ago"99.999% starting... now."
- devonbleak 14y agoThat's not how SLAs work. Basically they're guaranteeing 99.999% uptime across some interval and you get some kind of compensation if/when they fail to meet it.
- mootothemax 14y agoI have to wonder how many extra customers the various third-party DNS services have gained as a direct result of this. I've just switched to DNSMadeEasy - for anyone concerned about the time involved, they have some cool timesavers like templates you can apply to all of your domains at once. Really makes a difference not having to manually set up individually the entries for Google Apps on 20+ domains.
- tubbo 14y agoI used to think GoDaddy went down through no active fault of their own. Now I realize they're just a shitty company with a shitty product. Good riddance. Our company's email is hosted on GoDaddy, and it has the most downtime of any email service I use (which are Google Apps/Gmail, Yahoo for Business and a private IMAP server). Frequently it will just be dead in the middle of the day, which is why this particular outage wasn't so surprising to us.
- SilasX 14y agoSemi-OT: Why is it so hard to find yesterday's highly-rated GoDaddy outage discussion? Neither sorting by relevance nor recency nor points will find it. Or maybe there wasn't one?
- mongol 14y agoI don't know why it is hard but I picked it up from my browser history: http://news.ycombinator.com/item?id=4500993 http://news.ycombinator.com/item?id=4500993
- SilasX 14y agoYour confirmation of my sanity is appreciated :-)
- EwanToo 14y agosomeone has hidden it, no idea why though.
- lhnn 14y agohnsearch.com. I love it.
- kevincennis 14y agoFor anyone interested, the person who claimed responsibility for this is tweeting about GoDaddy's response: https://twitter.com/AnonymousOwn3r/status/245568841160196096 https://twitter.com/AnonymousOwn3r/status/245568841160196096
- stef25 14y agoI'm amazed at how many seemingly non technical people are cheering this guy on, or least seem to think it's "cool" what he's doing.
- zbuc 14y agoThis guy seems to be full of crap to me. Did you see the Pastebin he posted? "cool wow godaddy is so safe lol i just found 51 vulnerabilities http://pastebin.com/EdCDaqQy http://pastebin.com/EdCDaqQy " It's a bunch of form POST values he got 302 redirects back from? No indication of XSS at all. What's this dude going on about?
- dkopack 14y agoJust checked his twitter feed, looks like he's attempting to claim source from an old open source project as part of the claimed hack...credibility gone.
- goldeneye 14y agoThis statement makes a lot of sense. I found it a bit suspicious that Anonymous Own3r twitted: "When i do some DDOS attack i like to let it down by many days, the attack for unlimited time, it can last one hour or one month" Which sounds like he actually has no control over what is happening and makes a statement that is impossible to disprove.
- wethesheeple 14y ago"yet did not result in bringing other DNS services backup" Can you be more specific? Which other domain names did you try? Also, I believe some parts of the world were unaffected by the outage. I would guess a large majority of GoDaddy customers would not even know this outage occurred. They are "casual" domain name registrants and in some cases "casual" website operators. They registered some names and then never did anything with them. Or they operate a website but it's very low traffic and they rarely think about it. That is only a guess.
- oasisbob 14y agoOn the outages mailing list[1], Mike Dob (GoDaddy Network Engineering Manager) just added more details, saying: > It was BGP related and more details should be posted today [1] http://puck.nether.net/mailman/listinfo/outages http://puck.nether.net/mailman/listinfo/outages
- napillo 14y agoBGP? So it was a bad route injected? They should have had someone there looking for that, that certainly wouldn't take them down for 5 hours.
- dingdingpop 14y agoTheir engineer claims it was an issue with BGP (http://permalink.gmane.org/gmane.org.operators.isotf.outages/4279 http://permalink.gmane.org/gmane.org.operators.isotf.outages...). BGPlay (http://bgplay.routeviews.org/ http://bgplay.routeviews.org/) does not show anything indicative in the BGP default-free table (what the Internet sees), as abnormal or misconfigured. While there could be iBGP issues, like others have stated there was (intermittent) connectivity by IP during the outage. It's both bullshit PR and more importantly spreading disinformation to save face. Why? A security breach would instill customer fear and generate negative press. Customers would leave by the droves. A DoS/DDoS displays that GoDaddy has inadequate infrastructure while competitors such as CloudFare actually do. Furthermore, why would a company that pisses off the Internet be appealing to anyone? Again it will generate negative/bad press, and customers will leave by the drove. Spreading disinformation by claiming it was either a human error or equipment fault? From a company perspective this is actually the best option. Just provide generous service credit to your customers, you may generate positive press, you will gain customer goodwill and regain their confidence. This is GoDaddy's best option. Until they provide actual details with proof that it was a misconfiguration or hardware fault, I will continue to call bullshit. Too many factors don't add up, especially the publicly available data which monitors the BGP DFT on the Internet. The two conjectures that seem plausible so far is the SQL injection in their web interface for DNS and/or a DoS/DDoS attack.