5 ms·

Wow! Killing DNS is a nasty way to take down the site. That could take hours to come back.

• takeda64 14y ago
  Not really. Looks like their domain is configured so negative responses are cached only for 300 seconds (5 minutes).
  • insaneirish 14y ago
    TTLs are a recommendation, at best. Not worth relying on.
    • Florin_Andrei 14y ago
      Technically true. In practice it does/could work pretty well. Source: I've seen pure DNS-based site failover and it never had a problem.
      • bigiain 14y ago
        True. We did a fair bit of experimenting with this (for both load balancing and failover) a few years back, and so long as you stay at 300sec or greater it works fine for well over 99% of all the traffic we tested with. Once you dropped below 300 seconds problems started appearing - from memory some older versions of Windows would default to 900secs, 3600secs, or 86400secs. I haven't revisited that research for 3 or 4 years, but our findings then led to a policy of "if you're prepared to accept ~15mins of mixed availability (after you've identified a problem and hit the panic button) then DNS based failover works well enough. If you need significantly better response times than that, you need proper hardware/network based failover"
        • Florin_Andrei 14y ago
          Using 10sec TTL currently for site failover. No complaints.
          • bigiain 14y ago
            Do you have much Windows XP traffic? From memory, we saw evidence of XP using 86400sec (1 day) TTLs if you tried publishing TTLs lower than 300. (This was back in '08, so perhaps things have changed since. I'm still seeing ~30% WinXP in some of my Google Analytics accounts...)
    • TallGuyShort 14y ago
      Exactly - my biggest complaint about ELB on AWS is that they relied on people honouring a TTL that was ridiculously low, and as such now honoured by many clients.
      • sveiss 14y ago
        Yup. The day I got a bunch of Netflix traffic sent to servers I was running thanks to ELB and caching resolvers not honoring TTL values was great fun. (That was also the day we learnt that our 500 page was actually really expensive to render, which added to the excitement...)
• Silhouette 14y ago
  Given that they've been having can't-see-blogs-at-all scale problems for something like 24 hours already (I was looking for one last night, and it's currently nearly midnight UK time) I suspect a few more hours to clear up whatever fundamental screw-up is behind this could be a good investment.
  • nwh 14y ago
    Seeing as they have engineers deploying code directly into production with `vim`, a fundamental screwup was inevitable.
    • IgorPartola 14y ago
      No build process? In 2012? Do you have a reference for deploying code with vim?
      • nwh 14y ago
        The source of their index.php, and the subsequent discussion on HN. Just imagine someone franticly trying to get into insert mode, and saving on a production server. It stayed like that for a reasonable period before someone noticed. http://pastebin.com/raw.php?i=aPQJUh1Q http://pastebin.com/raw.php?i=aPQJUh1Q http://news.ycombinator.com/item?id=2343330 http://news.ycombinator.com/item?id=2343330
        • fryguy 14y ago
          That doesn't necessarily mean that they don't use source control or build control. The index file linked has tons of production-specific configuration information (database ip/keys), which wouldn't be checked into source control under best practices. That would require manual editing on the server to set up correctly.
          • bashtoni 14y ago
            s/manual editing/mangement from configuration management like Puppet or Chef/
• adrr 14y ago
  Recovery is going to spread over a few hours which could be a benefit.