4 ms·

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be miss

> It reduces alert fatigue by classifying alerts as actionable or noisy and providing contextual information for handling alerts. grimace face I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and we can certainly use more competition in this space, but this aspect of it is basically enabling bad cultural practices and I wouldn’t highlight it as a main selling point.

aray07 2y ago
Yeah, thats fair feedback. The main aim was to reduce the alert fatigue for on-call engineers and provide a way to get insight into the alerts at the end of the on-call shift. This way there is data to make a case that certain alerts are noisy (for various reasons) and we should strive to reduce the time spent dealing with these alerts. Fixing some of them might be as easy as deleting them but for others might need dedicated time working on them.
deleted 2y ago
[deleted]
DanHulton 2y ago
I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order. However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers. I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.
- steve1977 2y ago
  If you work at a shitty place, focus your energy on leaving the shitty place.
  - jgraettinger1 2y ago
    Or, it can be self-evident to all that today's work can deliver more benefit if focused elsewhere, instead of root-causing fickle alert flakes. Customers buy products and services, not alert hygiene.
  - karamanolev 2y ago
    The shittiness of a place is defined by a lot more, and more important, attributes than alert hygiene. Culture, pay, location, industry, leadership. If one leaves companies for (relatively) minor things like that, there are basically no companies left to work for.
    - steve1977 2y ago
      Alert hygiene is a symptom of bad culture. Not the only one of course.
stitched2gethr 2y ago
If I may try to counter this point. You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as such, and more context without using brain power is always welcome. I think I'll give it a shot.
- aray07 2y ago
  Thanks for the feedback. Yeah, ideally, teams would go back and fix their misconfigured alerts. Unfortunately, on-call ends and people forget. The aim was to both provide context when an alert comes up as well as provide a report at the end so everybody has context into the state of alerts.
unethical_ban 2y ago
If the tool works properly, then the value proposition is: Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.
- callalex 2y ago
  If something that triggers a pager takes months to resolve you already work at an organization that is so ossified it will be unwilling to adopt a random bandaid startup product like this.
  - unethical_ban 2y ago
    I'm sure that's true in many places. The number of large companies deal with this kind of thing (understaffed teams operating hundreds or thousands of devices) is quite high. Some can pull off shadow IT or exceptions for free software.
renger6002 2y ago
Agreed. This doesn't make the problem better, it's a bandaid solution that can make the problem worse by allowing you to ignore it for longer. Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.
- CPLX 2y ago
  Bandaids are a valuable and useful product used billions of times around the globe every year.
  - deleted 2y ago
    [deleted]
devjab 2y ago
As someone who has worked in non-tech startup transitioning into enterprise, or enterprise organisarions for decades… I can tell you that non of these companies or organisations are capable of making meaningful interactions with operations. Disclaimer I haven’t worked operations, but I’ve been in relative close proximity with them work wise for natural reasons or using their infrastructure and sometimes making their tools or helping them with things like automation and Powershell. Anyway in lot of places management see IT as a necessary evil. Like a service center similar to HR but less popular because management genuinely don’t understand it and most IT departments lack HRs political shrewdness and communication abilities. At the same time it’s not uncommon for users to be unable to tell support if they’re on an Android or iOS device (yes, I’m serious). Sometimes employees won’t even differentiate between their professional and personal IT issues on their work devices. Which means that sometimes they’ll raise hell over things that might not warrant full alert systems for on-site support. What might be challenging here is that you’ll still need someone to actually use the authors tool correctly. Though that is probably going to be a lot easier than making any sort of change management to was organisations relationship with IT.
Too 2y ago
100x this. Garbage in = Garbage out. In similar mindset, I've seen attempts to "fix" flaky test suites by retrying failing tests 5 times until they pass. What happens: You just set a new baseline of shit allowed. This allows even more noise to enter the system and you have to rerun them even more or need increasingly more advanced tools to filter out the noise. Once the new baseline is anchored, you become dependent on the filter. Now every tool that interacts with the metrics need to be aware of the additional filter, there may be more than just the slack messages. Should your dashboard show the raw or the filtered metrics? Devils advocate: consider an alert with 99/100 false positives. The LLM may be good at classifying it as noisy but will it do a better job than a human to react to the 1 true positive? Maybe, but at the same time it allows more such noise to accumulate in the system, in effect a net negative. It's better to remove such an alert instead. Even if the numbers were turned around in favor, that's a lot of complexity added. The additional context this product provides may of course still be useful and i applaud the effort. This product space does have a lot of potential for growth and is a real pain for operators. Be careful with using it as a substitute for proper alert hygiene and culture.
gklitz 2y ago
> enabling bad cultural practices I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have unintended consequences. Imagine a developer telling management that they didn’t implement error checking on some edge case because the support team told them they didn’t have documentation about how to take action for instance. The appropriate response there would be “why on earth are you asking support permission to add error messages for a known error?”. On the other hand, if a support team is drowning in noisy error messages they need tooling to make it easy to distinguish between those and other messages that need to be reviewed of have action taken.
- remus 2y ago
  > There is nothing culturally bad in a system issuing an error if there is an error. That's true, but if the error says "PANIC! EVERYTHING IS DOWN" when it's not true, then it's asking for an action that's outsized to the problem. Error messages are fine, but they just need to be classified and responded to correctly, and noisy alerts are typically the ones that are misclassified and demanding attention they (probably) don't deserve.
- madeofpalk 2y ago
  The context here is alerts triggering on-call. If the error is not-actionable, why wake someone up in the middle of the night because of it? I don't think anyone is rejecting the observability of these errors, but just that there's no point in having it alert/wake someone up unnecessarily.
- everforward 2y ago
  Then it either shouldn't be an alert (and instead part of some kind of summary report or some such) or the devs need to take on call. It is an exercise in frustration for everyone to route the page to ops just to make ops call dev; that means dev still has to have an oncall rotation, they might as well just take the page directly. The unintended consequence of forcing alerts down ops' throat is them gradually caring less about pages, because there's a very good chance that each one is unactionable. I've worked places that do this, I've seen it happen first-hand more than once. It starts with frustration and ops being less helpful to devs, and ends in a jaded acceptance where ops people start telling each other "just close it and see if it happens a second or third time, that alert never means anything". At that point, the system may as well not emit the errors anyways because no one is looking at the alerts anyways.
remus 2y ago
To give a more charitable take, in practice it's easy for noisy alerts to creep in and a tool like this could be a good nudge to dig a bit deeper on that alert and why it's gone from useful to noise.
williamdclt 2y ago
> which upon investigation might not even be “just” noise My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes. The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.
- stackskipton 2y ago
  Ops person here, fixing it is generally impossible as most alarms are due to code and development teams don't give a shit. "It's just one false alarm occasionally, what's the big deal?" or "We are no longer working on that product." or "I have 14 features to do, go away ops and deal." If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring. So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.
- bongodongobob 2y ago
  A lot of times ops can't fix it (false positive), it's another teams responsibility. And if that other team can't or won't fix it, ops is screwed with constant false positives. At that point, it's in ops best interest to ignore it, let the actual positive wreak havoc and point at the other team. If they don't have the capability of putting pressure on that team, sometimes a fire is the only way to do it. I'm not saying this is a good idea, but bureaucracy is going to bureaucracy. You have to make the other team feel your pain.
- aray07 2y ago
  Yeah, I was surprised at how common this behavior has been across companies. I don't have a good solution for this problem - I am hoping additional visibility and accountability into these alerts can be useful. One of the things we have been hoping to do is to be able to add user impact context into the alert enrichment as well.
zbentley 2y ago
Missing from the context of sibling replies is the (in my experience as an SRE, quite large) category of alerts that are mostly-but-not-entirely noise and high-effort-duration to properly improve. Consider a not-that-hypothetical example: "host computer is unreachable" alerts that page oncall when they arrive for members of a fleet of critical database servers or replicas. The alerts have proven their usefulness (they tend to arrive several minutes before application-level error spikes when a database is e.g. so overloaded the monitoring agent can't function or a replica is gone so changelogs are overflowing) ... when they're genuine. However, they're mostly not genuine: alerting agents crash and automatic-restart-service init scripts bug out or give up; per-database-owner customizations in hosts' available file descriptor numbers are propagated incorrectly to non-database services and prevent the alerting agent from running, databases that serve infrequent-but-critical on-demand reporting loads are subjected to tens-of-minutes-long load spikes during which the host is doing what it's supposed to but so pegged that the alerting agent won't work, and so on. What do you do with those alerts? "Just fix the problems causing the false positives!" Fine, but that takes a lot of time and coordinated effort, even if the oncall folks are empowered to prioritize the work getting done (which is far from a given at many companies, for reasons both good and bad): auto-restart-agent scripts can be replaced with better scripts (time, effort, debugging of some hokey bash that needs to run on a wide variety of environments) or systemd (time, effort, maintenance windows, and approval/retraining to update ancient linux distributions running critical databases). File descriptor/per-database tunings can be unified and continually audited for/invalidated before configs are pushed (developer effort, coordination with teams writing configs). Reporting databases can be upsized (money, maitenance windows) or the database processes can be moved into a cgroup to leave some resources to spare (effort, distro upgrades, maintenance windows). That's going to take awhile, if it ever happens to completion. Meanwhile, this "host unreachable" alert is useless 90% of the time and very useful (as in: it can be leveraged to prevent downtime for customers entirely) the remaining 10% of the time. Like, sure, some of those issues are stupid. But none are hypothetical, all are younger than 5y, and I bet this kind of struggle is common and representative even at companies who are invested in operations and operations staff. That's not an "inability to create useful observability", that's a genuinely hard problem resulting in noisy, spurious alerts that, depending on the rate-of-change/regulatory space of the company, might persist for months or years. What's more, alert management is an ongoing process. Even if one family of noisy alerts is addressed, another one will emerge as new behaviors and technologies are adopted. I guess this is all to say that I don't think tools like Opslane (which I have not used) are "enabling bad cultural practices". Organizations that don't give a shit about operations will continue to suck at operations no matter what tools they use. But products Opslane are valuable even (especially?) in capable, operations-focused organizations as well.
adamredwoods 2y ago
Agreed. To solve noisy alerts, we have a group channel where we 1. call it out 2. solve it with weekly meetings. We usually "group" up and fix it, so everyone is in agreement. It takes about an hour to organize everyone and talk through it, but it improves the on-call quality-of-life over time.

aray07 2y ago

Yeah, thats fair feedback. The main aim was to reduce the alert fatigue for on-call engineers and provide a way to get insight into the alerts at the end of the on-call shift. This way there is data to make a case that certain alerts are noisy (for various reasons) and we should strive to reduce the time spent dealing with these alerts. Fixing some of them might be as easy as deleting them but for others might need dedicated time working on them.

deleted 2y ago

[deleted]

DanHulton 2y ago

I agree with your intent and desire, but the fact is that this problem _keeps happening,_ and we can't fix it by advocating "well, just do alerts better." There's a lot of cultural inertia at a lot of places that leads to creating too many, too low-signal alerts, and fixing that across an entire company -- or, hell, industry -- is a magnificently tall order. However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers. I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.

steve1977 2y ago
If you work at a shitty place, focus your energy on leaving the shitty place.
- jgraettinger1 2y ago
  Or, it can be self-evident to all that today's work can deliver more benefit if focused elsewhere, instead of root-causing fickle alert flakes. Customers buy products and services, not alert hygiene.
- karamanolev 2y ago
  The shittiness of a place is defined by a lot more, and more important, attributes than alert hygiene. Culture, pay, location, industry, leadership. If one leaves companies for (relatively) minor things like that, there are basically no companies left to work for.
  - steve1977 2y ago
    Alert hygiene is a symptom of bad culture. Not the only one of course.

stitched2gethr 2y ago

If I may try to counter this point. You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as such, and more context without using brain power is always welcome. I think I'll give it a shot.

aray07 2y ago
Thanks for the feedback. Yeah, ideally, teams would go back and fix their misconfigured alerts. Unfortunately, on-call ends and people forget. The aim was to both provide context when an alert comes up as well as provide a report at the end so everybody has context into the state of alerts.

unethical_ban 2y ago

If the tool works properly, then the value proposition is: Sure, you could spend months trying to fix or tune out every useless alarm type, or try to hack your alert manager/email inbox with filters for things you know will get fixed in a few months - OR, you can use this tool that can quickly classify things as important or not.

callalex 2y ago
If something that triggers a pager takes months to resolve you already work at an organization that is so ossified it will be unwilling to adopt a random bandaid startup product like this.
- unethical_ban 2y ago
  I'm sure that's true in many places. The number of large companies deal with this kind of thing (understaffed teams operating hundreds or thousands of devices) is quite high. Some can pull off shadow IT or exceptions for free software.

renger6002 2y ago

Agreed. This doesn't make the problem better, it's a bandaid solution that can make the problem worse by allowing you to ignore it for longer. Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.

CPLX 2y ago
Bandaids are a valuable and useful product used billions of times around the globe every year.
- deleted 2y ago
  [deleted]

devjab 2y ago

As someone who has worked in non-tech startup transitioning into enterprise, or enterprise organisarions for decades… I can tell you that non of these companies or organisations are capable of making meaningful interactions with operations. Disclaimer I haven’t worked operations, but I’ve been in relative close proximity with them work wise for natural reasons or using their infrastructure and sometimes making their tools or helping them with things like automation and Powershell. Anyway in lot of places management see IT as a necessary evil. Like a service center similar to HR but less popular because management genuinely don’t understand it and most IT departments lack HRs political shrewdness and communication abilities. At the same time it’s not uncommon for users to be unable to tell support if they’re on an Android or iOS device (yes, I’m serious). Sometimes employees won’t even differentiate between their professional and personal IT issues on their work devices. Which means that sometimes they’ll raise hell over things that might not warrant full alert systems for on-site support. What might be challenging here is that you’ll still need someone to actually use the authors tool correctly. Though that is probably going to be a lot easier than making any sort of change management to was organisations relationship with IT.

Too 2y ago

100x this. Garbage in = Garbage out. In similar mindset, I've seen attempts to "fix" flaky test suites by retrying failing tests 5 times until they pass. What happens: You just set a new baseline of shit allowed. This allows even more noise to enter the system and you have to rerun them even more or need increasingly more advanced tools to filter out the noise. Once the new baseline is anchored, you become dependent on the filter. Now every tool that interacts with the metrics need to be aware of the additional filter, there may be more than just the slack messages. Should your dashboard show the raw or the filtered metrics? Devils advocate: consider an alert with 99/100 false positives. The LLM may be good at classifying it as noisy but will it do a better job than a human to react to the 1 true positive? Maybe, but at the same time it allows more such noise to accumulate in the system, in effect a net negative. It's better to remove such an alert instead. Even if the numbers were turned around in favor, that's a lot of complexity added. The additional context this product provides may of course still be useful and i applaud the effort. This product space does have a lot of potential for growth and is a real pain for operators. Be careful with using it as a substitute for proper alert hygiene and culture.

gklitz 2y ago

> enabling bad cultural practices I strongly disagree. There is nothing culturally bad in a system issuing an error if there is an error. Sometimes systems issue errors that are considered noise by supporters because they are not actionable, but forcing a system to not issue an error just because your support team cannot directly take action on it is an extremely odd leakage of team responsibilities and bound to have unintended consequences. Imagine a developer telling management that they didn’t implement error checking on some edge case because the support team told them they didn’t have documentation about how to take action for instance. The appropriate response there would be “why on earth are you asking support permission to add error messages for a known error?”. On the other hand, if a support team is drowning in noisy error messages they need tooling to make it easy to distinguish between those and other messages that need to be reviewed of have action taken.

remus 2y ago
> There is nothing culturally bad in a system issuing an error if there is an error. That's true, but if the error says "PANIC! EVERYTHING IS DOWN" when it's not true, then it's asking for an action that's outsized to the problem. Error messages are fine, but they just need to be classified and responded to correctly, and noisy alerts are typically the ones that are misclassified and demanding attention they (probably) don't deserve.
madeofpalk 2y ago
The context here is alerts triggering on-call. If the error is not-actionable, why wake someone up in the middle of the night because of it? I don't think anyone is rejecting the observability of these errors, but just that there's no point in having it alert/wake someone up unnecessarily.
everforward 2y ago
Then it either shouldn't be an alert (and instead part of some kind of summary report or some such) or the devs need to take on call. It is an exercise in frustration for everyone to route the page to ops just to make ops call dev; that means dev still has to have an oncall rotation, they might as well just take the page directly. The unintended consequence of forcing alerts down ops' throat is them gradually caring less about pages, because there's a very good chance that each one is unactionable. I've worked places that do this, I've seen it happen first-hand more than once. It starts with frustration and ops being less helpful to devs, and ends in a jaded acceptance where ops people start telling each other "just close it and see if it happens a second or third time, that alert never means anything". At that point, the system may as well not emit the errors anyways because no one is looking at the alerts anyways.

remus 2y ago

To give a more charitable take, in practice it's easy for noisy alerts to creep in and a tool like this could be a good nudge to dig a bit deeper on that alert and why it's gone from useful to noise.

williamdclt 2y ago

> which upon investigation might not even be “just” noise My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes. The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.

stackskipton 2y ago
Ops person here, fixing it is generally impossible as most alarms are due to code and development teams don't give a shit. "It's just one false alarm occasionally, what's the big deal?" or "We are no longer working on that product." or "I have 14 features to do, go away ops and deal." If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring. So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.
bongodongobob 2y ago
A lot of times ops can't fix it (false positive), it's another teams responsibility. And if that other team can't or won't fix it, ops is screwed with constant false positives. At that point, it's in ops best interest to ignore it, let the actual positive wreak havoc and point at the other team. If they don't have the capability of putting pressure on that team, sometimes a fire is the only way to do it. I'm not saying this is a good idea, but bureaucracy is going to bureaucracy. You have to make the other team feel your pain.
aray07 2y ago
Yeah, I was surprised at how common this behavior has been across companies. I don't have a good solution for this problem - I am hoping additional visibility and accountability into these alerts can be useful. One of the things we have been hoping to do is to be able to add user impact context into the alert enrichment as well.

zbentley 2y ago

Missing from the context of sibling replies is the (in my experience as an SRE, quite large) category of alerts that are mostly-but-not-entirely noise and high-effort-duration to properly improve. Consider a not-that-hypothetical example: "host computer is unreachable" alerts that page oncall when they arrive for members of a fleet of critical database servers or replicas. The alerts have proven their usefulness (they tend to arrive several minutes before application-level error spikes when a database is e.g. so overloaded the monitoring agent can't function or a replica is gone so changelogs are overflowing) ... when they're genuine. However, they're mostly not genuine: alerting agents crash and automatic-restart-service init scripts bug out or give up; per-database-owner customizations in hosts' available file descriptor numbers are propagated incorrectly to non-database services and prevent the alerting agent from running, databases that serve infrequent-but-critical on-demand reporting loads are subjected to tens-of-minutes-long load spikes during which the host is doing what it's supposed to but so pegged that the alerting agent won't work, and so on. What do you do with those alerts? "Just fix the problems causing the false positives!" Fine, but that takes a lot of time and coordinated effort, even if the oncall folks are empowered to prioritize the work getting done (which is far from a given at many companies, for reasons both good and bad): auto-restart-agent scripts can be replaced with better scripts (time, effort, debugging of some hokey bash that needs to run on a wide variety of environments) or systemd (time, effort, maintenance windows, and approval/retraining to update ancient linux distributions running critical databases). File descriptor/per-database tunings can be unified and continually audited for/invalidated before configs are pushed (developer effort, coordination with teams writing configs). Reporting databases can be upsized (money, maitenance windows) or the database processes can be moved into a cgroup to leave some resources to spare (effort, distro upgrades, maintenance windows). That's going to take awhile, if it ever happens to completion. Meanwhile, this "host unreachable" alert is useless 90% of the time and very useful (as in: it can be leveraged to prevent downtime for customers entirely) the remaining 10% of the time. Like, sure, some of those issues are stupid. But none are hypothetical, all are younger than 5y, and I bet this kind of struggle is common and representative even at companies who are invested in operations and operations staff. That's not an "inability to create useful observability", that's a genuinely hard problem resulting in noisy, spurious alerts that, depending on the rate-of-change/regulatory space of the company, might persist for months or years. What's more, alert management is an ongoing process. Even if one family of noisy alerts is addressed, another one will emerge as new behaviors and technologies are adopted. I guess this is all to say that I don't think tools like Opslane (which I have not used) are "enabling bad cultural practices". Organizations that don't give a shit about operations will continue to suck at operations no matter what tools they use. But products Opslane are valuable even (especially?) in capable, operations-focused organizations as well.

adamredwoods 2y ago

Agreed. To solve noisy alerts, we have a group channel where we 1. call it out 2. solve it with weekly meetings. We usually "group" up and fix it, so everyone is in agreement. It takes about an hour to organize everyone and talk through it, but it improves the on-call quality-of-life over time.