3 ms·
Telecoms solved this problem fifteen years ago when they started automating Fault Management (google it). Granted, neural networks were not generally applicabl
by Jolter 2y ago
Telecoms solved this problem fifteen years ago when they started automating Fault Management (google it).
Granted, neural networks were not generally applicable to this problem at the time, but this whole idea seems like the same problem being solved again.
Telecoms and IT used to supervise their networks using Alarms, in either a Network Management System (NMS) or something more ad-hoc like Nagios. There, you got structured alarms over a network, like SNMP traps, that got stored as records in a database. It’s fairly easy to program filters using simple counting or more complex heuristics against a database.
Now, for some reason, alerting has shifted to Slack. Naturally since the data is now unstructured text, the solution involves an LLM! You build complexity into the filtering solution because you have an alarm infrastructure that’s too simple.
- samcat116 2y agoThe alerts being sent to Slack are normally from one of those alert databases (such as Prometheus and AlertManager). Slack isn't the source of truth for them, just a notification channel.
- Jolter 2y agoOh, Prometheus is good for metrics but it doesn’t hold alarms in the Fault Management sense, though. It only keeps the metrics and thresholds, checks for threshold violations, and then alerts via some mechanism. If it were an alarm database, an operator would be able to 1. Acknowledge the alarm 2. Manually clear an alarm that was issued in error. Without those mechanisms, alarm handling becomes really difficult for an ops team, because now all you have is either a string of emails or a chat log.
- drivers99 2y agoThe wikipedia for page for fault management has a "see also" for alarm management, which looks extremely relevant as well.