2 ms·
So I once brought down an alerting system using Excel (btw, this story is more about unintended consequences instead of MSFT) - I own an alerting system - Fo
by alexpotato 1y ago
So I once brought down an alerting system using Excel
(btw, this story is more about unintended consequences instead of MSFT)
- I own an alerting system
- For log based alerts, it looks for a keyword e.g. "alert_log"
- I make a spreadsheet to track data about alerts and call one of the sheets "alert_log"
- Alert system starts going crazy: using tons of CPU, number of alerts processed goes through the roof but not a lot of alerts generated
- Turns out that I was using the cloud version of Excel so any text entered transited the firewall
- Firewall logs store the text "alert_log"
- Alert system thinks it's an alert BUT it's not a real alert so triggers an alert processing alert
- That second alert contains the text from the firewall log and so cycle begins
In other words, systems can operate in weird ways and then cause things to happen you didn't anticipate. It's why things like audits, red teaming and defense in depth all matter.
- unethical_ban 1y agoAs a firewall engineer I have to tell people to make sure to disable traffic logs for syslogs from the firewall for this reason.
- _whiteCaps_ 1y agoReminds me of the time I set up tcpdump to log network traffic on a troublesome server. To save disk space I sent it over SSH to my laptop. Oops!