8 ms·
When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will
by buro9 3y ago
When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data.
The lifecycle is: PoPs generate/gather data > send to PDX > compute in PDX > ship updates / data to PoPs.
If you take out PDX, then as so much runs on fresh data, it starts getting stale.
I doubt everything has changed since then, so this is unlikely just "API down" and more likely that a lot of things are now in a degraded state as they're running on stale information (no update from PDX)... this includes things like load balancing, the tiered caching (Argo Smart Routing), Warp / Zero Trust, etc.
Even if it were only "API down", then bear in mind that a lot of automation customers have will block attacks by calling the API... "API down" is a hell of a window of opportunity for attackers.
Note that just before I'd left they'd been investing in standing up AMS (I think) but had never successfully tested a significant failover, and the majority of services that needed fresh state did not know how to do this.
PS: :scream: most of the observability was also based in PDX, so hugs to all the teams and SREs currently running blind.
- flutas 3y agoSomeone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. [0]: https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149...
- supriyo-biswas 3y agoThe more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.
- dharmab 3y agoKubernetes' default datastore, etcd, is not tolerant of latencies between multiple regions. Generally, vanilla k8s clusters have a single-region control plane.
- supriyo-biswas 3y agoThis can just be multiple datacenters located close together (~100 km) similar to AWS AZs.
- dharmab 3y agoFun fact, on certain (major) cloud providers, in certain regions, AZs are sometimes different floors of the same building :)
- mh- 3y agoYou may be obligated not to name them, but I'm not: Google.
- slig 3y agoThe latency is exceptional.
- pests 3y agoI like how clear Azure is on this: > Availability zones are unique physical locations within an Azure region. Each zone is made up of one or more datacenters with independent power, cooling, and networking. The physical separation of availability zones within a region limits the impact to applications and data from zone failures, such as power and cooling failures, large-scale flooding, major storms and superstorms, and other events that could disrupt site access, safe passage, extended utilities uptime, and the availability of resources. https://learn.microsoft.com/en-us/azure/architecture/high-availability/building-solutions-for-high-availability https://learn.microsoft.com/en-us/azure/architecture/high-av...
- gemstones 3y agoI think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
- nik736 3y agoEven the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.
- milkglass 3y agoCost of that likely is ginormous compared to their SLA obligations.
- thedaly 3y agoThe phone utility were I live has deisel generators that kick on whenever the power goes out in order to keep the copper phone lines operational. These generators always work, or at least one of the four they have in each office does.
- bombcar 3y agoThe datacenter I was in for awhile had the big gens, and with similar "phone utility" setups - they would cut to the backup gens once a month and run for longer than the UPS could hold the facility (if they detected an issue, they'd switch back to utility power). They also had redundant gensets (2x the whole facility, 4x 'important stuff' - you could get a bit of a discount by being willing to be shut off in a huge emergency where gens were dying/running out of fuel).
- jrockway 3y agoI wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop. Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.
- marcc 3y agoYikes. If still true, this feels like a significant single point of failure in their architecture.
- pornel 3y agoIIRC there is supposed to be a failover to AMS.
- deleted 3y ago[deleted]
- 015a 3y ago> Note that just before I'd left they'd been investing in standing up AMS (I think) but had never successfully tested a significant failover IIRC, AMS was/is just a new DC to replace or augment LUX, which has existed for some time as, among other things, a failover secondary for PDX. But yeah, intention and reality drifted, as they always do; I had heard a variety of reasons why automated, or even break-glass, failover to a DC on the opposite side of the planet wasn't a tenable solution for many workloads. They're big on chaos testing, and definitely (very regularly) test whole-DC network partitions. I'd love to be a fly on the wall in the meeting discussing why this event was so different.
- roamerz 3y agoGenerators, AC units, UPS’s and auto cutover switches are all mechanical susceptible to failure. You can do your best with maintenance and pre planning but in the end there is a non zero chance that a condition that you didn’t anticipate or couldn’t afford to account for will happen and your critical infrastructure will go down.
- ilaksh 3y agoWhat is PDX?
- nosequel 3y agoCloudflare's Portland datacenter. Most clouds/CDNs name their DC's after the airport code of the city they are in or are close to. Internally they would be listed as PDX, ORD, LAX, etc.
- pard68 3y agoPortland Data Center I believe
- jeffwask 3y agoI assume shorthand for a datacenter in Portland. Labeling data centers by their cities airport code is something I see a lot.
- kbknapp 3y agoIt's the IATA code for Portland International Airport. Many datacenters use IATA codes for the nearest airport to give a rough approximation of the location. So the PDX datacenter is the one closest to the PDX airport in or around Portland.
- anonymoushn 3y agoAs of a few years ago, tiered cache and Argo are different things. I only wrote one of them, but in the marketing they are always referred to together. I think other than ENT users who can customize their tiered cache topology using the API it should be totally fine for the tiered cache topology to be days or weeks out of date. A lot of other stuff will have trouble, as you've correctly noted.
- simple10 3y agoDNS updates were also down during a good chuck of the outage. Most likely due to degraded API access since I'm guessing the dashboard UI relies on the API to make updates. But even when DNS updates managed to update in the UI, they still weren't propagating. I had to delete entire domains and recreate them after the outage in Cloudflare dashboard in order for SSL and DNS propagation to work again.