4 ms·

The incident report is very light on details. For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remo

The incident report is very light on details. For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers or how they'd contain it in the future, except for that they'd need to "fail fast" in this situation. (You can obviously catch it with a blue-green deployment, which they do mention, but at that point some small portion of traffic has already been affected.)

jgrahamc 4y ago
This incident report was out within hours of the failure. The team is now doing an internal retro to look at decision speed, rollback speed, and the circumstances around that function having a side effect. We'll update the post with more detail.
- cesarb 4y ago
  > We'll update the post with more detail. Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.
stall84 4y ago
Yes they probably can fill in plenty of detail, but overall I was impressed with the incident report on it's own.. I don't think it's super common for companies' their size to release a timeline/report on the fkup unless they're coerced to.
dspillett 4y ago
> For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers They probably don't need to at all, but I can think or several ways it might unintentionally happen. Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should. How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!

jgrahamc 4y ago

This incident report was out within hours of the failure. The team is now doing an internal retro to look at decision speed, rollback speed, and the circumstances around that function having a side effect. We'll update the post with more detail.

cesarb 4y ago
> We'll update the post with more detail. Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.

stall84 4y ago

Yes they probably can fill in plenty of detail, but overall I was impressed with the incident report on it's own.. I don't think it's super common for companies' their size to release a timeline/report on the fkup unless they're coerced to.

dspillett 4y ago

> For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers They probably don't need to at all, but I can think or several ways it might unintentionally happen. Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should. How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!