5 ms·

Googler here - my opinions are my own, not representing the company at the lowest level in case of severe outage we resort to IRC, Plain Old Telephone Service

• jug 5y ago
  Some people here say their fallback IRC doesn't work due to DNS reliance. :|
  • lmitfb 5y ago
    That would completely defeat the purpose... I have a hard time believing that.
    • jaywalk 5y ago
      Why? Even if it's not DNS reliance, if they self-hosted the server (very likely) then it'll be just as unreachable as everything else within their network at the moment.
      • yupper32 5y ago
        The entire purpose of an IRC backup is in case shit hits the fan. That means having it run on a completely separate stack. What use is it if it runs on the same stack as what you might be trying to fix?
        • jaywalk 5y ago
          Clearly "our entire network is down, worldwide" wasn't part of their planning. Don't get too cocky with your 20/20 hindsight.
          • yupper32 5y ago
            I don't think it's cocky or 20/20 hindsight. Companies I've worked for specifically set up IRC in part because "our entire network is down, worldwide" can happen and you need a way to communicate.
            • nl 5y ago
              I bet they never tested taking out their own DNS. IRC does use DNS at least to get hostnames during connection. I'd be surprised if it didn't use it at other points.
              • Retric 5y ago
                I’ve setup hosts files in case DNS was down to access critical systems before. It’s a perfectly reasonable precaution.
                • edoceo 5y ago
                  My small org, maybe 50 ips/hosts we care about, maintain a hosts file stills, for those nodes public and internal names. It's in Git, spread around and we also have our fingers crossed.
      • littlecranky67 5y ago
        If only IRC would have been built with multi-server setups in mind, that forward messages between servers, and continues to work if a single - or even a set - of servers would go down, just resulting in a netsplit...Oh wait, it was! My bet is, FB will reach out to others in FAMANG, and an interest group will form maintaining such an emergency infrastructure comm network. Basically a network for network engineers. Because media (and shareholders) will soon ask Microsoft and Google what their plans for such situations are. I'm very glad FB is not in the cloud business...
        • rrix2 5y ago
          > If only IRC would have been built with multi-server setups in mind, that forward messages between servers, and continues to work if a single - or even a set - of servers would go down, just resulting in a netsplit...Oh wait, it was! yeah if only Facebook's production engineering team had hired a team of full time IRCops for their emergency fallback network...
          • littlecranky67 5y ago
            Considering how much IRCops were paid back in the day (mostly zero as they were volunteers) and what a single senior engineer at FB makes, I'm sure you will find 3-4 people spread amongst the world willing to share this 250k+ salary amongst them.
        • ceva 5y ago
          That is called outbound network :)
  • comonoid 5y ago
    One of my employers once forced all the staff to use an internally-developed messenger (for sake of security, but some politics was involved as well), but made an exception for the devops team who used Telegram.
    • wut42 5y ago
      Telegram? Interesting choice!
      • dikei 5y ago
        Devops like Telegram because it has proper bot API, unlike many other competitors.
        • wut42 5y ago
          Oh! It makes sense. While I don't like telegram for some reasons, their API is totally top notch and a real pleasure to work with.
• Johnny555 5y ago
  Around here we use Slack for primary communications, Google Hangouts (or Chat or whatever they call it now) as secondary, and we keep an on-call list with phone numbers in our main Git repo, so everyone has it checked out on their laptop, so if the SHTF, we can resort to voice and/or SMS. I remembered to publish my cell phone's real number on the on-call list rather than just my Google Voice number since if Hangouts is down, Google Voice might be too.
  • texasbigdata 5y ago
    Where are the tapes though? Colo on separate tectonic tape or nah?
    • Johnny555 5y ago
      ?
      • tijtij 5y ago
        I think texasbigdata is talking about backup tapes and maybe mistyped tectonic plate Backup tapes and in production servers are kept at different colocation sites to protect data from fire and other catastrophes of that level Using colo sites on separate tectonic plates would protect you from catastrophes on a geological cataclysm level
        • Johnny555 5y ago
          We don't use tapes, everything we have is in the cloud, at a minimum everything is spread over multiple datacenters (AZ's in AWS parlance), important stuff is spread over multiple regions, or depending on the data, multiple cloud providers. Last time I used tape, we used Ironmountain to haul the tapes 60 miles away which was determined to be far enough for seismic safety, but that was over a decade ago.
        • texasbigdata 5y ago
          Thank you kind sir.
• guidoism 5y ago
  I worked on the identity system that chat (whatever the current name is) and gmail depend on and we used IRC since if we relied on the system we support we wouldn’t be able to fix it.