3 ms·

A lot of these questions bely a misunderstanding of how it works - bot management is evaluated inline within the proxy as a feature on the site (similar to othe

A lot of these questions bely a misunderstanding of how it works - bot management is evaluated inline within the proxy as a feature on the site (similar to other features like image optimization). So during ingress there’s not an async call to the bot management service which intercepts the request before it’s outbound to origin - it’s literally a Lua script (or rust module in fl2) that runs on ingress inline as part of handling the request. Thus there’s no timeout or other concerns with the management service failing to assign a bot score. There are better questions but to me the ones posed don’t seem particularly interesting.

kqr 1y ago
Maybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!
- tptacek 1y ago
  I don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is some other technical change, maybe the issue is human/procedural. How could you possibly know? The owners of the system probably don't have a full answer within just a few hours of handling the incident!
  - kqr 1y ago
    I would be worried if they had all the answers within a few hours! I was just caught off guard by the focus on technical control measures when there seems to have been fairly obvious problems with information channels. For example, "more global kill switches for features" is good, but would "only" have shaved 30 % off the time of recovery (if reading the timeline charitably). Being able to identify the broken component faster would have shaved 30–70 % off the time of recovery depending on how fast identification could happen – even with no improvements to the kill switch situation.
- vlovich123 1y ago
  It’s literally in the same process as the router. Performance issues like the layer taking too long are handled differently than just timing out afaik. It’s possible it should time out but safe cancellation in Rust can be tricky depending on how this is architected. Timeouts are irrelevant to this outage and I wouldn’t classify it as a huge issue unless there’s a possibility that a layer has an indefinite blocking call as a misbehavior and forever hangs all requests. My hunch is that this is generally already handled - FL2 is not documented enough to speculate.

kqr 1y ago

Maybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!

tptacek 1y ago
I don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is some other technical change, maybe the issue is human/procedural. How could you possibly know? The owners of the system probably don't have a full answer within just a few hours of handling the incident!
- kqr 1y ago
  I would be worried if they had all the answers within a few hours! I was just caught off guard by the focus on technical control measures when there seems to have been fairly obvious problems with information channels. For example, "more global kill switches for features" is good, but would "only" have shaved 30 % off the time of recovery (if reading the timeline charitably). Being able to identify the broken component faster would have shaved 30–70 % off the time of recovery depending on how fast identification could happen – even with no improvements to the kill switch situation.
vlovich123 1y ago
It’s literally in the same process as the router. Performance issues like the layer taking too long are handled differently than just timing out afaik. It’s possible it should time out but safe cancellation in Rust can be tricky depending on how this is architected. Timeouts are irrelevant to this outage and I wouldn’t classify it as a huge issue unless there’s a possibility that a layer has an indefinite blocking call as a misbehavior and forever hangs all requests. My hunch is that this is generally already handled - FL2 is not documented enough to speculate.