3 ms·

> Here's a 2 minute demo of what it looks like: https://youtu.be/t83Egs5l8ok https://youtu.be/t83Egs5l8ok. The problem with this demo is that it uses something

by treis 4y ago

> Here's a 2 minute demo of what it looks like: https://youtu.be/t83Egs5l8ok https://youtu.be/t83Egs5l8ok. The problem with this demo is that it uses something that's 100% broken due to something that happened immediately before the failure. That's not hard to debug and I don't really see value there. The scenarios that could use this sort of tool are things like someone turning on a flag that breaks 1% of a specific end point but only 0.1% of overall requests. So something sub-alert level and with not nearly an immediately obvious cause & effect. If you can detect something like that without generating a ton of noise and give a hint to root cause then that'd be something killer. It's a cool idea and I can see the value. We've had scenarios like the one I mentioned (and worse) go undetected because of the noise Sentry generates. If you can solve that then you've really got something.

netfortius 4y ago
>"... I don't really see value there ... I can see the value." Huh!?!
- treis 4y ago
  They're referring to different things....
stochastimus 4y ago
Hey! Founder here. Thanks for your comment! > That's not hard to debug I agree there's a large class of problems where that's the case, and I also agree it's perhaps easier than many of the issues we've caught in the real world, such as are discussed in the Cisco case study. But I still want to defend the example as non-trivial. So of course we pick up the errors in services that are fall-out from the breakage, and we pick up the chaos test running, which leaves a pretty wide footprint. But if something like this happened in the real world, the important event would have been that one that's in the RC report, that's not even an error (though it is quite unusual): the kernel log entry pointing out the eth0 misconfig. It can take a long time for someone to get around to poring through their various host kernel logs and looking at everything, even non-errors, so having it surfaced to you right away feels like a very useful thing. At the same time though, I like the example you gave even more. What many people will do is upload their own incident data via the CLI, or deploy our chart into a staging environment and just break things, to see what happens. Based on your description, there's a good chance we would pick it up since we track anomalies and perform anomaly correlation across all pairs of streams: we're not looking for a threshold percentage of overall error rates, for example, anywhere. If you'd be willing to help us test this use-case, I'd love to work with you personally to help you get up-and-running. Also, as we see more and more failure modes, we continue to make our detection algorithms more robust. While we generally achieve >>90% detection of incidents and their root cause indicators overall, there are always places we can and would love to do better. I think you'd find that our solution would catch most of what you'd want out-of-the-box, and that we're responsive enough to learn from every customer. Lastly, I would ask: do you think that we should do videos of some more "subtle" examples (for lack of a better word)? As someone viewing the website, would you have watched them? Thanks again for your feedback!

by treis 4y ago

netfortius 4y ago

>"... I don't really see value there ... I can see the value." Huh!?!

treis 4y ago
They're referring to different things....

stochastimus 4y ago

Hey! Founder here. Thanks for your comment! > That's not hard to debug I agree there's a large class of problems where that's the case, and I also agree it's perhaps easier than many of the issues we've caught in the real world, such as are discussed in the Cisco case study. But I still want to defend the example as non-trivial. So of course we pick up the errors in services that are fall-out from the breakage, and we pick up the chaos test running, which leaves a pretty wide footprint. But if something like this happened in the real world, the important event would have been that one that's in the RC report, that's not even an error (though it is quite unusual): the kernel log entry pointing out the eth0 misconfig. It can take a long time for someone to get around to poring through their various host kernel logs and looking at everything, even non-errors, so having it surfaced to you right away feels like a very useful thing. At the same time though, I like the example you gave even more. What many people will do is upload their own incident data via the CLI, or deploy our chart into a staging environment and just break things, to see what happens. Based on your description, there's a good chance we would pick it up since we track anomalies and perform anomaly correlation across all pairs of streams: we're not looking for a threshold percentage of overall error rates, for example, anywhere. If you'd be willing to help us test this use-case, I'd love to work with you personally to help you get up-and-running. Also, as we see more and more failure modes, we continue to make our detection algorithms more robust. While we generally achieve >>90% detection of incidents and their root cause indicators overall, there are always places we can and would love to do better. I think you'd find that our solution would catch most of what you'd want out-of-the-box, and that we're responsive enough to learn from every customer. Lastly, I would ask: do you think that we should do videos of some more "subtle" examples (for lack of a better word)? As someone viewing the website, would you have watched them? Thanks again for your feedback!