3 ms·
I do security consulting and that very situation is common. I know several BigCorps that specifically turned off Splunk alerts because it was too noisy. Some of
by txcwpalpha 8y ago
I do security consulting and that very situation is common. I know several BigCorps that specifically turned off Splunk alerts because it was too noisy. Some of them had valid reasoning for it, other didn't.
There's also no guarantee that if they did have the Splunk alerts enabled, that they would have caught anything. In one instance I can think of, having Splunk alerts enabled did more harm than good because it meant SOC staff were busy chasing down hundreds of false alerts (aka the "noise" in "too noisy") and missed real threats. With the alerts off, the staff could use other more reliable sources of threat notifications and catch more of the real threats.
Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy".
- outworlder 8y ago> Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy". At some point, the size of your staff doesn't really matter, unless it's at roughly number of alerts / 2. Now not only you have the alerting noise, but you also have the added noise of the big intra-team communication. Cleaning up those alerts needs to be first priority. Not to mention, usually such a barrage of alerts means they are very poorly designed.
- Aeolun 8y agoI don’t get it. Can’t you tweak the settings on these alerts until they’re effective never sent during normal operations? I mean, that might take you a day, maybe a week to catch the edge cases, and then it’s not a problem any more...
- jeieo3949 8y agoOh yeah, these tools are all trainable In my experiences though, there’s no time to become an expert anymore You get it set up, then a PM or manager with no clue redirects us to the next priority. Meanwhile the new monitoring tool just hangs out being underutilized at $20k/mo Not to say all PMs or managers are clueless, but there are more clueless ones out there than not. They are noise generators themselves. Or, IMO worse, the sycophant coders who concern themselves with their own personal metrics, release dumpster fire after dumpster fire. And us in devops and secops are fighting their idiocy all day, not training tools to catch intruders. I’ve worked with far too many people who epitomize that meme of Captain Kirk where he can’t hear you over how awesome he is
- ISL 8y agoTurning off alerts is a quick way to ensure that they are effectively never sent during normal operations. Most days, it is not incorrect.
- Spare_account 8y ago> Most days, it is not incorrect This feels a bit like saying a stopped clock tells the correct time twice a day. Turning off the alerts doesn't cause a problem unless there was something worth alerting about that day.
- 616c 8y agoAnd how did you get into this type of consulting?