5 ms·

I know for a fact that they also had Splunk ES bought, built, and running, then the CISO had them turn off the alerts because they were "too noisy" and sure eno

by oglopf 8y ago

I know for a fact that they also had Splunk ES bought, built, and running, then the CISO had them turn off the alerts because they were "too noisy" and sure enough, they could see all this happening when it was retroactively inspected. There are so many things wrong in that sentence but my most glaring question is: why on earth was she even receiving the alerts? That's so far below her level. So, there's more here than just the cert issue. There's also just a complete disregard for a secure mindset from the person that is supposed to be setting that mindset in the culture, along with what sounds like micro-managing to a state of no-managing.

rkho 8y ago
The incredibly frightening thing is that I'm no longer surprised about the fact that people like these still remain completely employable.
- richforrester 8y ago
  Unfortunately, management roles are often inherited, instead of being earned.
rjzzleep 8y ago
It's very likely that she comes from corporate and thinks she has to know everything. I worked for a guy like that. He wanted everything on his plate and simple things like a confluence update could take 1.5 years because of it.
barbecue_sauce 8y ago
Depends on how big their team is and its structure with regard to alerting responsibility. One would expect one of the big credit rating agencies to have a large IT team, but old school corporations view IT as a cost center and like to run skeleton crews (which can flatten out administration hierarchies).
- CreatedAccount 8y ago
  I work for one of the big three and we generally are pretty lean when it comes to staffing. Our CEO likes to claim it benefits us because then they don't have to do big layoffs during downturns.
  - zentiggr 8y ago
    "Can't chop off heads if there aren't enough to start with" .... now there's MBA logic for you.
  - adrianmonk 8y ago
    Well, if there are zero consequences to failure, why spend money preventing it? Maybe it does benefit companies in that position.
txcwpalpha 8y ago
I do security consulting and that very situation is common. I know several BigCorps that specifically turned off Splunk alerts because it was too noisy. Some of them had valid reasoning for it, other didn't. There's also no guarantee that if they did have the Splunk alerts enabled, that they would have caught anything. In one instance I can think of, having Splunk alerts enabled did more harm than good because it meant SOC staff were busy chasing down hundreds of false alerts (aka the "noise" in "too noisy") and missed real threats. With the alerts off, the staff could use other more reliable sources of threat notifications and catch more of the real threats. Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy".
- outworlder 8y ago
  > Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy". At some point, the size of your staff doesn't really matter, unless it's at roughly number of alerts / 2. Now not only you have the alerting noise, but you also have the added noise of the big intra-team communication. Cleaning up those alerts needs to be first priority. Not to mention, usually such a barrage of alerts means they are very poorly designed.
- Aeolun 8y ago
  I don’t get it. Can’t you tweak the settings on these alerts until they’re effective never sent during normal operations? I mean, that might take you a day, maybe a week to catch the edge cases, and then it’s not a problem any more...
  - jeieo3949 8y ago
    Oh yeah, these tools are all trainable In my experiences though, there’s no time to become an expert anymore You get it set up, then a PM or manager with no clue redirects us to the next priority. Meanwhile the new monitoring tool just hangs out being underutilized at $20k/mo Not to say all PMs or managers are clueless, but there are more clueless ones out there than not. They are noise generators themselves. Or, IMO worse, the sycophant coders who concern themselves with their own personal metrics, release dumpster fire after dumpster fire. And us in devops and secops are fighting their idiocy all day, not training tools to catch intruders. I’ve worked with far too many people who epitomize that meme of Captain Kirk where he can’t hear you over how awesome he is
jstarfish 8y ago
> I know for a fact that they also had Splunk ES bought, built, and running, then the CISO had them turn off the alerts because they were "too noisy" and sure enough, they could see all this happening when it was retroactively inspected. Splunk (ES or not) is not a magic bullet. It generates a lot of bullshit to justify its expense. Finding the signal in the noise is easy when you know what the signature of the largest data breach in history was supposed to look like. What exactly does this prove? > Why on earth was she even receiving the alerts? That's so far below her level. Knowing the C-levels there, they likely wanted earlier warning of emerging incidents-- EFX and their subsidiaries had been dealing with a steady stream of incidents (some public, some not) for a while before the big one. Krebs covered some of them.
outworlder 8y ago
I wrote a lengthy response on how suppressing false alarms is absolutely the right thing to do, then realized that the CISO was the one requesting it. That really makes no sense. The engineers should be the ones determining which ones to adjust and which to suppress. Middle managers are already too far removed from the productive chain to even make the determination if an alert is false or not, let alone C-level.
- dmix 8y ago
  > Middle managers are already too far removed from the productive chain to even make the determination if an alert is false or not, let alone C-level. This is assuming the security team was big enough to have that many layers to create such a disconnect...
- deleted 8y ago
  [deleted]
jiveturkey 8y ago
< then the CISO had them turn off the alerts I'm reminded of the Target breach.
duxup 8y ago
An Equifax exec went to Parana Bread (of all places) and later had trouble even understanding what a security researcher told them they had an API with customer data that was easily found and queried from the internet. The executive got confused and suspicious when the security researcher offered to send additional data and talked about sending that data securely via PGP (it seems like he simply didn't know what the request for a PGP key... even meant). https://arstechnica.com/information-technology/2018/04/panera-accused-security-researcher-of-scam-when-he-reported-a-major-flaw/ https://arstechnica.com/information-technology/2018/04/paner... It would seem a lot of Equifax executives are happy to act on their own in spite of their own ignorance. I know a handful of folks in security, they run into complete morons in the security industry all the time. It's scary how wildly variable the talent in the industry is at all levels. To some extent that talent variability is everywhere, but the sheer volume of people are so clueless that they could be described as a liability in some situations is surprisingly high when it comes to security. I know security folks who have changed jobs simply because they didn't want to be associated with someone who took over at their employer. Reputation among the folks who know seems to be a big deal to many of them.
- Salgat 8y ago
  Oh I remember this. Dumbass had no idea what kind of issue he had on his hands and just blew it off not realizing the level of blowback and publicity that would occur if the whitehat decided to publicly announce it. Boggles the mind how these people get such high paying jobs.
  - duxup 8y ago
    It is so confusing as the pattern of: 1. Dude contacts you. 2. You fail to work with him / or just don't. 3. It becomes public and you look like idiots. That pattern is so established that every security executive should know you want to avoid that... but nope, some run headlong into it. I'm no security dude at all, and I can recognize that pattern just from reading the news. What on earth is an executive reading if they have no idea?
    - 8y ago

by oglopf 8y ago

rkho 8y ago

The incredibly frightening thing is that I'm no longer surprised about the fact that people like these still remain completely employable.

richforrester 8y ago
Unfortunately, management roles are often inherited, instead of being earned.

rjzzleep 8y ago

It's very likely that she comes from corporate and thinks she has to know everything. I worked for a guy like that. He wanted everything on his plate and simple things like a confluence update could take 1.5 years because of it.

barbecue_sauce 8y ago

Depends on how big their team is and its structure with regard to alerting responsibility. One would expect one of the big credit rating agencies to have a large IT team, but old school corporations view IT as a cost center and like to run skeleton crews (which can flatten out administration hierarchies).

CreatedAccount 8y ago
I work for one of the big three and we generally are pretty lean when it comes to staffing. Our CEO likes to claim it benefits us because then they don't have to do big layoffs during downturns.
- zentiggr 8y ago
  "Can't chop off heads if there aren't enough to start with" .... now there's MBA logic for you.
- adrianmonk 8y ago
  Well, if there are zero consequences to failure, why spend money preventing it? Maybe it does benefit companies in that position.

txcwpalpha 8y ago

I do security consulting and that very situation is common. I know several BigCorps that specifically turned off Splunk alerts because it was too noisy. Some of them had valid reasoning for it, other didn't. There's also no guarantee that if they did have the Splunk alerts enabled, that they would have caught anything. In one instance I can think of, having Splunk alerts enabled did more harm than good because it meant SOC staff were busy chasing down hundreds of false alerts (aka the "noise" in "too noisy") and missed real threats. With the alerts off, the staff could use other more reliable sources of threat notifications and catch more of the real threats. Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy".

outworlder 8y ago
> Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy". At some point, the size of your staff doesn't really matter, unless it's at roughly number of alerts / 2. Now not only you have the alerting noise, but you also have the added noise of the big intra-team communication. Cleaning up those alerts needs to be first priority. Not to mention, usually such a barrage of alerts means they are very poorly designed.
Aeolun 8y ago
I don’t get it. Can’t you tweak the settings on these alerts until they’re effective never sent during normal operations? I mean, that might take you a day, maybe a week to catch the edge cases, and then it’s not a problem any more...
- jeieo3949 8y ago
  Oh yeah, these tools are all trainable In my experiences though, there’s no time to become an expert anymore You get it set up, then a PM or manager with no clue redirects us to the next priority. Meanwhile the new monitoring tool just hangs out being underutilized at $20k/mo Not to say all PMs or managers are clueless, but there are more clueless ones out there than not. They are noise generators themselves. Or, IMO worse, the sycophant coders who concern themselves with their own personal metrics, release dumpster fire after dumpster fire. And us in devops and secops are fighting their idiocy all day, not training tools to catch intruders. I’ve worked with far too many people who epitomize that meme of Captain Kirk where he can’t hear you over how awesome he is

jstarfish 8y ago

> I know for a fact that they also had Splunk ES bought, built, and running, then the CISO had them turn off the alerts because they were "too noisy" and sure enough, they could see all this happening when it was retroactively inspected. Splunk (ES or not) is not a magic bullet. It generates a lot of bullshit to justify its expense. Finding the signal in the noise is easy when you know what the signature of the largest data breach in history was supposed to look like. What exactly does this prove? > Why on earth was she even receiving the alerts? That's so far below her level. Knowing the C-levels there, they likely wanted earlier warning of emerging incidents-- EFX and their subsidiaries had been dealing with a steady stream of incidents (some public, some not) for a while before the big one. Krebs covered some of them.

outworlder 8y ago

I wrote a lengthy response on how suppressing false alarms is absolutely the right thing to do, then realized that the CISO was the one requesting it. That really makes no sense. The engineers should be the ones determining which ones to adjust and which to suppress. Middle managers are already too far removed from the productive chain to even make the determination if an alert is false or not, let alone C-level.

dmix 8y ago
> Middle managers are already too far removed from the productive chain to even make the determination if an alert is false or not, let alone C-level. This is assuming the security team was big enough to have that many layers to create such a disconnect...
deleted 8y ago
[deleted]

jiveturkey 8y ago

< then the CISO had them turn off the alerts I'm reminded of the Target breach.

duxup 8y ago

An Equifax exec went to Parana Bread (of all places) and later had trouble even understanding what a security researcher told them they had an API with customer data that was easily found and queried from the internet. The executive got confused and suspicious when the security researcher offered to send additional data and talked about sending that data securely via PGP (it seems like he simply didn't know what the request for a PGP key... even meant). https://arstechnica.com/information-technology/2018/04/panera-accused-security-researcher-of-scam-when-he-reported-a-major-flaw/ https://arstechnica.com/information-technology/2018/04/paner... It would seem a lot of Equifax executives are happy to act on their own in spite of their own ignorance. I know a handful of folks in security, they run into complete morons in the security industry all the time. It's scary how wildly variable the talent in the industry is at all levels. To some extent that talent variability is everywhere, but the sheer volume of people are so clueless that they could be described as a liability in some situations is surprisingly high when it comes to security. I know security folks who have changed jobs simply because they didn't want to be associated with someone who took over at their employer. Reputation among the folks who know seems to be a big deal to many of them.

Salgat 8y ago
Oh I remember this. Dumbass had no idea what kind of issue he had on his hands and just blew it off not realizing the level of blowback and publicity that would occur if the whitehat decided to publicly announce it. Boggles the mind how these people get such high paying jobs.
- duxup 8y ago
  It is so confusing as the pattern of: 1. Dude contacts you. 2. You fail to work with him / or just don't. 3. It becomes public and you look like idiots. That pattern is so established that every security executive should know you want to avoid that... but nope, some run headlong into it. I'm no security dude at all, and I can recognize that pattern just from reading the news. What on earth is an executive reading if they have no idea?
  - 8y ago