7 ms·

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assess

• supriyo-biswas 3y ago
  The more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.
  • dharmab 3y ago
    Kubernetes' default datastore, etcd, is not tolerant of latencies between multiple regions. Generally, vanilla k8s clusters have a single-region control plane.
    • supriyo-biswas 3y ago
      This can just be multiple datacenters located close together (~100 km) similar to AWS AZs.
      • dharmab 3y ago
        Fun fact, on certain (major) cloud providers, in certain regions, AZs are sometimes different floors of the same building :)
        • mh- 3y ago
          You may be obligated not to name them, but I'm not: Google.
          • dharmab 3y ago
            That actually wasn't the one I was thinking of!
          • matja 3y ago
            AZ is a term used by AWS and Azure. GCP documentation makes it clear to "Distribute your resources across multiple zones and regions", where regions are physically different data centers.
        • slig 3y ago
          The latency is exceptional.
        • pests 3y ago
          I like how clear Azure is on this: > Availability zones are unique physical locations within an Azure region. Each zone is made up of one or more datacenters with independent power, cooling, and networking. The physical separation of availability zones within a region limits the impact to applications and data from zone failures, such as power and cooling failures, large-scale flooding, major storms and superstorms, and other events that could disrupt site access, safe passage, extended utilities uptime, and the availability of resources. https://learn.microsoft.com/en-us/azure/architecture/high-availability/building-solutions-for-high-availability https://learn.microsoft.com/en-us/azure/architecture/high-av...
          • dharmab 3y ago
            I think they expanded Tokyo but previously that was a single-building "region" And it's virtually impossible to make, say, a Singapore region resilient to natural disasters
            • pests 3y ago
    • __turbobrew__ 3y ago
      We run a k8s control plane across datacenter in west, central, and east US and it works fine.
      • dharmab 3y ago
        I assume your site to site latency's under 100ms? If so that's fine.
• gemstones 3y ago
  I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
  • nik736 3y ago
    Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.
    • milkglass 3y ago
      Cost of that likely is ginormous compared to their SLA obligations.
    • thedaly 3y ago
      The phone utility were I live has deisel generators that kick on whenever the power goes out in order to keep the copper phone lines operational. These generators always work, or at least one of the four they have in each office does.
      • bombcar 3y ago
        The datacenter I was in for awhile had the big gens, and with similar "phone utility" setups - they would cut to the backup gens once a month and run for longer than the UPS could hold the facility (if they detected an issue, they'd switch back to utility power). They also had redundant gensets (2x the whole facility, 4x 'important stuff' - you could get a bit of a discount by being willing to be shut off in a huge emergency where gens were dying/running out of fuel).
    • jrockway 3y ago
      I wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop. Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.