7 ms·

Tumblr just dropped out of DNS

133 points by state_machine 14y ago

deleted 14y ago
[deleted]
Pr0 14y ago
Seems it started with just a normal outage: http://thenextweb.com/insider/2012/12/12/tumblr-confirms-users-are-experiencing-slow-loading-and-intermittent-errors-engineers-are-on-it/ http://thenextweb.com/insider/2012/12/12/tumblr-confirms-use...
emeraldd 14y ago
dig +trace www.tumblr.com *snip* tumblr.com. 300 IN SOA pdns1.ultradns.net. hostmaster.tumblr.com. 2012121602 86400 7200 604800 300 ;; Received 108 bytes from 204.74.108.1#53(204.74.108.1) in 21 ms Looks like their dns is down alright. You could try in your hosts file: 72.32.231.8 www.tumblr.com tumblr.com The ip is from their whois info and appears to be giving the We're sorry error message.
- rtkwe 14y ago
  Huh the only way you can tell 72.32.231.8 redirects to tumblr's crash page is the link under find out why. Completely unbranded.
kondro 14y ago
Could someone refresh my memory on how Tumblr makes its revenue?
- sjtgraham 14y ago
  If you really want to know why don't you find out and then come back to tell us instead of making fatuous comments
  - kondro 14y ago
    Love to, but Tumblr's site is down… so I can't check.
- citricsquid 14y ago
  Sponsored posts, premium themes, promoted posts. Pretty sure they're far from profitable, but they're bringing some revenue in. http://www.quora.com/Tumblr/How-does-Tumblr-make-money http://www.quora.com/Tumblr/How-does-Tumblr-make-money http://www.businessinsider.com/tumblr-revenues-2012-9 http://www.businessinsider.com/tumblr-revenues-2012-9
  - cft 14y ago
    If they are not profitable, it means that the downtime is actually improving their financials...;)
    - ChuckMcM 14y ago
      Chuckled at that, given their scale I'm sure they are already at fixed rates for colo space and bandwidth.
- jonathanmoore 14y ago
  - Brand spotlights - Sponsored radar posts (ads) - Promoted posts (for all users) - Premium theme shared revenue - Selling access to data
tmx 14y ago
That's pretty bad. Admittedly, they update their zone very frequently (every time a user signs up/changes name/deletes themselves), but you'd think they would have an independent secondary DNS provider somewhere.
- cmelbye 14y ago
  Do they really? It seems more likely that they'd simply use a wildcard.
  - tmx 14y ago
    That'll teach me to post late at night. I forgot wildcards existed. I was thinking that (if I built tumblr) they would do subdomain searching at the DNS level to avoid hitting their database. Makes it even worse if they are using wildcards.
    - X-Istence 14y ago
      But if you don't publish an A record you get an ugly browser error message, whereas publishing the wildcard gives the user a notice saying that no such blog exists.
- rdl 14y ago
  I believe they use ultradns (a nice, but very expensive, dns service).
kordless 14y ago
CloudFlare. Seriously guys.
deleted 14y ago
[deleted]
- Maxious 14y ago
  -1 These notifications and the technical analysis that only HN provides are valuable for all us devop plebs trying to avoid the same thing.
  - skyebook 14y ago
    +1 for your -1, these notifications and the postmortems are always excellent sources of information
    - zachinglis 14y ago
      Compare it to the stuff that could be up here. People's projects. I'm not sure what's to learn from this other than: * Tumblr is down * Killing the DNS is bad
  - lhnn 14y ago
    There is no valuable postmortem on that instrusive-ad ridden site.
- zachinglis 14y ago
  +1 It's infuriating and seems like karma bait.
el_cuadrado 14y ago
Looks like they have MX and TXT, but missing A. Weird.
onetwothreefour 14y ago
Looks like UltraDNS disabled their zones. Someone probably forgot to pay invoices or something. You're pretty much screwed in this case.
- emeraldd 14y ago
  What are you basing the "disabled zone" statement on? A 'dig ANY tumblr.com' shows MX, TXT, and NS records . . .
sbierwagen 14y ago
@tumblr Tumblr has taken the site down in order to resolve a network issue. We will update as we know more. https://twitter.com/tumblr/status/279000741706878976 https://twitter.com/tumblr/status/279000741706878976
- emeraldd 14y ago
  Wow! Killing DNS is a nasty way to take down the site. That could take hours to come back.
  - takeda64 14y ago
    Not really. Looks like their domain is configured so negative responses are cached only for 300 seconds (5 minutes).
    - insaneirish 14y ago
      TTLs are a recommendation, at best. Not worth relying on.
      - Florin_Andrei 14y ago
        Technically true. In practice it does/could work pretty well. Source: I've seen pure DNS-based site failover and it never had a problem.
        bigiain 14y ago
        True. We did a fair bit of experimenting with this (for both load balancing and failover) a few years back, and so long as you stay at 300sec or greater it works fine for well over 99% of all the traffic we tested with. Once you dropped below 300 seconds problems started appearing - from memory some older versions of Windows would default to 900secs, 3600secs, or 86400secs. I haven't revisited that research for 3 or 4 years, but our findings then led to a policy of "if you're prepared to accept ~15mins of mixed availability (after you've identified a problem and hit the panic button) then DNS based failover works well enough. If you need significantly better response times than that, you need proper hardware/network based failover"
rdl 14y ago
They get hundreds of extra points for hosting their status page on their own service. That's basically the main (already widely accepted) lesson people should take from this -- people want twitter updates as well as an outside-hosted blog and monitor for service availability.
- Achshar 14y ago
  Doesn't serving status page form the site itself defeat the purpose of status page? In what case would that make any sense? I am now intrigued, this is not a small thing that can get over looked.
  - rdl 14y ago
    Yes, exactly. For status other than "site is down", it's probably nice to have information on-site since I guess tumblr is about sharing easily within tumblr. But maintenance and server status needs to be both on-site (tumblr.com/status ?) and an off-site page at status.tumblr.com.
  - oh_sigh 14y ago
    No, it does not defeat the purpose. Instead, it just lessens the effectiveness of the status page. It really depends on what tumblrs most common error scenario is. 95% of errors may manifest themselves in manners which do not affect the status of the status page.
    - deleted 14y ago
      [deleted]
  - moe 14y ago
    The whole notion of "hosting" a status page is ridiculous. You just generate it as static HTML and dump it on a bunch of free webhosters (or clouds if you want to be fancy). It's not rocket science to keep a HTML page online.
    - rdl 14y ago
brokentone 14y ago
Remember when we all decided Tumblr wasn't stable to host a professional site? Well... these people didn't. http://theatlantic.tumblr.com/ http://theatlantic.tumblr.com/ http://fox411.blogs.foxnews.com/ http://fox411.blogs.foxnews.com/ http://motherjones.tumblr.com/ http://motherjones.tumblr.com/ http://gq.tumblr.com/ http://gq.tumblr.com/ http://tumblr.elle.com/ http://tumblr.elle.com/
- blake8086 14y ago
  If the cost of hosting elsewhere exceeds the cost of the downtime, they made the right choice.
  - brokentone 14y ago
    Used to work at Fox, helped write the theme Lot lot lot more costs involved, including an inane templating system that took way longer to work with than to build normally. Hosting would have happened on the existing platform adding little to no cost. Also, the cost (lost revenue, people running around pointing fingers) of downtime for a site that large (yes, even the entertainment blog) quickly DWARFS any possible hosting costs. edit:formatting
- OverlordXenu 14y ago
  Aren't those essentially just advertising for the main sites?
state_machine 14y ago
Incidentally, this means every startup (and some of the bigger kids) that hosts its status page on Tumblr is missing a status page at the moment... eg: http://status.twitter.com http://status.twitter.com
- citricsquid 14y ago
  The Minecraft launcher also relies on tumblr to serve the landing page, a couple of million players will be seeing this: http://i.imgur.com/Zft71.png http://i.imgur.com/Zft71.png
  - Foomandoonian 14y ago
    We've seen that before. It's better than the skin servers being down. Which they probably are.
    - rtkwe 14y ago
      I'd rather no skins than the constant auth.minecraft outages.
- munificent 14y ago
  There's something beautiful about: * Tumbler reports its outage status using twitter. * Twitter reports its outage status using tumbler.
  - TallboyOne 14y ago
    Beautiful
bbuffone 14y ago
Visit this page to see the length of downtime -> http://www.websitetest.com/ui/tests/50c922b17a6c8757bb000005 http://www.websitetest.com/ui/tests/50c922b17a6c8757bb000005. The test will run every 10 minutes for the next 10 hours. Testing is only good for diagnosing issues like downtime and performance issues.
- cmark99 14y ago
  The monitoring shows that the site is back online around 02:46:09 GMT 12/13/2012 (9:46PM ET Dec 12th). When did the problem start? So this outage lasted 4 hours?
vhost- 14y ago
Speculation: Gmail, Facebook, then Tumblr? Kill switch? Half kidding...
- beagle3 14y ago
  I was having the same thought. But until proven otherwise, I'll assume strange coincidence. Although, when microsoft.com falls of the face of the earth tomorrow, I'm calling it a conspiracy.
cllns 14y ago
The main site/webapp is still down (3 hours later) but individual sites are up. http://theparisreview.tumblr.com/ http://theparisreview.tumblr.com/ They're returning 66.6.36.7 for DNS.

133 points by state_machine 14y ago

deleted 14y ago

[deleted]

Pr0 14y ago

Seems it started with just a normal outage: http://thenextweb.com/insider/2012/12/12/tumblr-confirms-users-are-experiencing-slow-loading-and-intermittent-errors-engineers-are-on-it/ http://thenextweb.com/insider/2012/12/12/tumblr-confirms-use...

emeraldd 14y ago

dig +trace www.tumblr.com *snip* tumblr.com. 300 IN SOA pdns1.ultradns.net. hostmaster.tumblr.com. 2012121602 86400 7200 604800 300 ;; Received 108 bytes from 204.74.108.1#53(204.74.108.1) in 21 ms Looks like their dns is down alright. You could try in your hosts file: 72.32.231.8 www.tumblr.com tumblr.com The ip is from their whois info and appears to be giving the We're sorry error message.

rtkwe 14y ago
Huh the only way you can tell 72.32.231.8 redirects to tumblr's crash page is the link under find out why. Completely unbranded.

kondro 14y ago

Could someone refresh my memory on how Tumblr makes its revenue?

sjtgraham 14y ago
If you really want to know why don't you find out and then come back to tell us instead of making fatuous comments
- kondro 14y ago
  Love to, but Tumblr's site is down… so I can't check.
citricsquid 14y ago
Sponsored posts, premium themes, promoted posts. Pretty sure they're far from profitable, but they're bringing some revenue in. http://www.quora.com/Tumblr/How-does-Tumblr-make-money http://www.quora.com/Tumblr/How-does-Tumblr-make-money http://www.businessinsider.com/tumblr-revenues-2012-9 http://www.businessinsider.com/tumblr-revenues-2012-9
- cft 14y ago
  If they are not profitable, it means that the downtime is actually improving their financials...;)
  - ChuckMcM 14y ago
    Chuckled at that, given their scale I'm sure they are already at fixed rates for colo space and bandwidth.
jonathanmoore 14y ago
- Brand spotlights - Sponsored radar posts (ads) - Promoted posts (for all users) - Premium theme shared revenue - Selling access to data

tmx 14y ago

That's pretty bad. Admittedly, they update their zone very frequently (every time a user signs up/changes name/deletes themselves), but you'd think they would have an independent secondary DNS provider somewhere.

cmelbye 14y ago
Do they really? It seems more likely that they'd simply use a wildcard.
- tmx 14y ago
  That'll teach me to post late at night. I forgot wildcards existed. I was thinking that (if I built tumblr) they would do subdomain searching at the DNS level to avoid hitting their database. Makes it even worse if they are using wildcards.
  - X-Istence 14y ago
    But if you don't publish an A record you get an ugly browser error message, whereas publishing the wildcard gives the user a notice saying that no such blog exists.
rdl 14y ago
I believe they use ultradns (a nice, but very expensive, dns service).

kordless 14y ago

CloudFlare. Seriously guys.

deleted 14y ago

[deleted]

Maxious 14y ago
-1 These notifications and the technical analysis that only HN provides are valuable for all us devop plebs trying to avoid the same thing.
- skyebook 14y ago
  +1 for your -1, these notifications and the postmortems are always excellent sources of information
  - zachinglis 14y ago
    Compare it to the stuff that could be up here. People's projects. I'm not sure what's to learn from this other than: * Tumblr is down * Killing the DNS is bad
- lhnn 14y ago
  There is no valuable postmortem on that instrusive-ad ridden site.
zachinglis 14y ago
+1 It's infuriating and seems like karma bait.

el_cuadrado 14y ago

Looks like they have MX and TXT, but missing A. Weird.

onetwothreefour 14y ago

Looks like UltraDNS disabled their zones. Someone probably forgot to pay invoices or something. You're pretty much screwed in this case.

emeraldd 14y ago
What are you basing the "disabled zone" statement on? A 'dig ANY tumblr.com' shows MX, TXT, and NS records . . .

sbierwagen 14y ago

@tumblr Tumblr has taken the site down in order to resolve a network issue. We will update as we know more. https://twitter.com/tumblr/status/279000741706878976 https://twitter.com/tumblr/status/279000741706878976

emeraldd 14y ago
Wow! Killing DNS is a nasty way to take down the site. That could take hours to come back.
- takeda64 14y ago
  Not really. Looks like their domain is configured so negative responses are cached only for 300 seconds (5 minutes).
  - insaneirish 14y ago
    TTLs are a recommendation, at best. Not worth relying on.
    - Florin_Andrei 14y ago
      Technically true. In practice it does/could work pretty well. Source: I've seen pure DNS-based site failover and it never had a problem.
      - bigiain 14y ago
        True. We did a fair bit of experimenting with this (for both load balancing and failover) a few years back, and so long as you stay at 300sec or greater it works fine for well over 99% of all the traffic we tested with. Once you dropped below 300 seconds problems started appearing - from memory some older versions of Windows would default to 900secs, 3600secs, or 86400secs. I haven't revisited that research for 3 or 4 years, but our findings then led to a policy of "if you're prepared to accept ~15mins of mixed availability (after you've identified a problem and hit the panic button) then DNS based failover works well enough. If you need significantly better response times than that, you need proper hardware/network based failover"

rdl 14y ago

They get hundreds of extra points for hosting their status page on their own service. That's basically the main (already widely accepted) lesson people should take from this -- people want twitter updates as well as an outside-hosted blog and monitor for service availability.

Achshar 14y ago
Doesn't serving status page form the site itself defeat the purpose of status page? In what case would that make any sense? I am now intrigued, this is not a small thing that can get over looked.
- rdl 14y ago
  Yes, exactly. For status other than "site is down", it's probably nice to have information on-site since I guess tumblr is about sharing easily within tumblr. But maintenance and server status needs to be both on-site (tumblr.com/status ?) and an off-site page at status.tumblr.com.
- oh_sigh 14y ago
  No, it does not defeat the purpose. Instead, it just lessens the effectiveness of the status page. It really depends on what tumblrs most common error scenario is. 95% of errors may manifest themselves in manners which do not affect the status of the status page.
  - deleted 14y ago
    [deleted]
- moe 14y ago
  The whole notion of "hosting" a status page is ridiculous. You just generate it as static HTML and dump it on a bunch of free webhosters (or clouds if you want to be fancy). It's not rocket science to keep a HTML page online.
  - rdl 14y ago

brokentone 14y ago

Remember when we all decided Tumblr wasn't stable to host a professional site? Well... these people didn't. http://theatlantic.tumblr.com/ http://theatlantic.tumblr.com/ http://fox411.blogs.foxnews.com/ http://fox411.blogs.foxnews.com/ http://motherjones.tumblr.com/ http://motherjones.tumblr.com/ http://gq.tumblr.com/ http://gq.tumblr.com/ http://tumblr.elle.com/ http://tumblr.elle.com/

blake8086 14y ago
If the cost of hosting elsewhere exceeds the cost of the downtime, they made the right choice.
- brokentone 14y ago
  Used to work at Fox, helped write the theme Lot lot lot more costs involved, including an inane templating system that took way longer to work with than to build normally. Hosting would have happened on the existing platform adding little to no cost. Also, the cost (lost revenue, people running around pointing fingers) of downtime for a site that large (yes, even the entertainment blog) quickly DWARFS any possible hosting costs. edit:formatting
OverlordXenu 14y ago
Aren't those essentially just advertising for the main sites?

state_machine 14y ago

Incidentally, this means every startup (and some of the bigger kids) that hosts its status page on Tumblr is missing a status page at the moment... eg: http://status.twitter.com http://status.twitter.com

citricsquid 14y ago
The Minecraft launcher also relies on tumblr to serve the landing page, a couple of million players will be seeing this: http://i.imgur.com/Zft71.png http://i.imgur.com/Zft71.png
- Foomandoonian 14y ago
  We've seen that before. It's better than the skin servers being down. Which they probably are.
  - rtkwe 14y ago
    I'd rather no skins than the constant auth.minecraft outages.
munificent 14y ago
There's something beautiful about: * Tumbler reports its outage status using twitter. * Twitter reports its outage status using tumbler.
- TallboyOne 14y ago
  Beautiful

bbuffone 14y ago

Visit this page to see the length of downtime -> http://www.websitetest.com/ui/tests/50c922b17a6c8757bb000005 http://www.websitetest.com/ui/tests/50c922b17a6c8757bb000005. The test will run every 10 minutes for the next 10 hours. Testing is only good for diagnosing issues like downtime and performance issues.

cmark99 14y ago
The monitoring shows that the site is back online around 02:46:09 GMT 12/13/2012 (9:46PM ET Dec 12th). When did the problem start? So this outage lasted 4 hours?

vhost- 14y ago

Speculation: Gmail, Facebook, then Tumblr? Kill switch? Half kidding...

beagle3 14y ago
I was having the same thought. But until proven otherwise, I'll assume strange coincidence. Although, when microsoft.com falls of the face of the earth tomorrow, I'm calling it a conspiracy.

cllns 14y ago

The main site/webapp is still down (3 hours later) but individual sites are up. http://theparisreview.tumblr.com/ http://theparisreview.tumblr.com/ They're returning 66.6.36.7 for DNS.