eastdakota

6 ms·

1.
by eastdakota 3mo ago
You’re confusing us with Google. We don’t have a visual CAPTCHA.
2.
by eastdakota 11mo ago
That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed t
3.
by eastdakota 11mo ago
There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing tha
4.
by eastdakota 11mo ago
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column n
5.
by eastdakota 11mo ago
* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.
6.
by eastdakota 11mo ago
Next time open your dev console in your window and look at how much is going on in the background.
7.
by eastdakota 11mo ago
We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world
8.
by eastdakota 11mo ago
Chicken burrito from Coyo Taco in Lisbon. I am not proud of this. It’s worse than ordering from Chipotle. But there are no Chipotle’s in Lisbon… yet.
9.
by eastdakota 11mo ago
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. S
10.
by eastdakota 11mo ago
Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.
11.
by eastdakota 11mo ago
That’s correct.
12.
by eastdakota 11mo ago
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad
13.
by eastdakota 11mo ago
We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.
14.
Cloudflare outage on November 18, 2025 post mortem(blog.cloudflare.com)
1465 points by eastdakota 11mo ago|916 comments
15.
by eastdakota 1y ago
That definitely is one not-wrong conclusion.
16.
by eastdakota 1y ago
Blog run by the engineering team. I wouldn’t even know how to veto a post if I wanted to. Not in our DNA.
17.
by eastdakota 1y ago
I’m not going anywhere anytime soon.
18.
Cloudflare's 2025 Annual Founders' Letter(blog.cloudflare.com)
16 points by eastdakota 1y ago|2 comments
19.
by eastdakota 1y ago
Free customers are served from the nearest location where we have capacity. If we’re capacity constrained then free customers will be the first to be rerouted to another facility with capacity. That typically only happens for a very narrow
20.
by eastdakota 1y ago
I agree with Kenton’s aside.
21.
by eastdakota 2y ago
We’re in Munich, pretty close, and Lisbon, which is lovely. Unlikely Austria any time soon.
22.
by eastdakota 2y ago
Had no idea there was a patent. Even if we had, think we’d have risked it.
23.
by eastdakota 2y ago
If it were merely marketing spend for customer acquisition, I bet the ROI on the lava lamp wall in SF has been 100,000x. This isn’t hard to figure out.
24.
by eastdakota 2y ago
Actually, it’s because we believe innovation is easier the further you are from HQ. And Portugal is wonderful. I’m spending about 4 months of the year there with the team who hail from around the world.
25.
by eastdakota 2y ago
No we’re not. We opened an India office because it’s a big market for us. But nothing has gotten “offshored” there.
26.
by eastdakota 2y ago
If you figure out how to model this fluid dynamics accurately over any reasonable period of time, call me. Lots and lots of more valuable things you could do with that, e.g., accurately predicting the weather.
27.
by eastdakota 2y ago
They should never all be off at the same time. We do cycle through each of them turning off for a period of the day. But, even if they were all off, there are lots of other sources of entropy we use, most of which are for more traditional i
28.
by eastdakota 2y ago
Mistakes inevitably happen at scale. Sometimes they’re not caught by traditional channels. What I try and encourage is our leaders to take responsibility and fix them wherever they see them. Which is what John did above.
29.
by eastdakota 2y ago
This is correct.
30.
by eastdakota 2y ago
Great thing about entropy is that adding more never hurts. This is one of many sources — both more conventional as well as unconventional — that we use. If it were to go offline, or somehow be corrupted, it wouldn’t hurt our ability to gene

More ›