8 ms·

Capital One Cyber Staff Raised Concerns Before Hack

141 points by valiant-comma 7y ago

ummonk 7y ago
https://www.linkedin.com/in/michael-johnson-098437117/ https://www.linkedin.com/in/michael-johnson-098437117/ They hired a bureaucrat, not an engineer, to be their CISO.
- hanniabu 7y ago
  Sounds like they hired an Olympian
- rficcaglia 7y ago
  He worked at Sandia so he seems to have engineering chops. But it is ridiculous to think that one individual or even one isolated group can defend against pervasive threats. Like worker safety, it has to be a cultural priority starting at the top and made priority #1 in every decision and work process. Even then, security is an illusion. Eventually something or someone will find a way to get through. Zero trust needs to be embraced. Assume every employee, partner, developer and vendor is going to have full internal access and abuse it...then defend from there.
- txcwpalpha 7y ago
  Hiring an engineer (and by that I assume you mean someone who has the bulk of their experience as a software engineer) as CISO is exactly the opposite of what a company should do. Security is absolutely not the same skill set as software engineering, and it's a huge misconception that people equate infosec with programming. Security engineers of course should be programmers, but an enormous amount of work that goes into infosec has absolutely nothing to do with code, and this is especially true for someone high up in management ranks like a CISO where their entire job is dealing with bureaucracy. Hiring an engineer to be a CISO would be like choosing a biologist to give you surgery instead of a surgeon. Yes, a biologist probably knows a lot about the nitty-gritty of how bodies work, but knowing how the body works and knowing how to perform surgery on a body are two very, very different things. That guy's LinkedIn looks more than qualified to be a CISO, and is certainly more qualified than the vast majority of CISOs I worked with in my career as a security consultant. And on top of all that, he has a degree in computer engineering and has prior experience as an engineer, so I have no idea why you're claiming he isn't an engineer.
  - goatinaboat 7y ago
    I see where you’re coming from but this is not far off the thinking that “an MBA can manage anything”. To use your analogy no surgeon is allowed to cut until he or she does have a solid grounding in biology.
    - txcwpalpha 7y ago
      You're right, that wasn't a very good analogy. My point is not to say that a CISO/surgeon shouldn't have any experience in engineering/biology (in fact, as you pointed out, it is the opposite! you ideally want a CISO with some technical chops just as you want a surgeon with some biology knowledge). My point is that there is much more to being a CISO than just technical knowledge. Maybe a better analogy would be choosing someone to represent you in a lawsuit about programming patents. You would want a lawyer representing you, not a programmer. Ideally the lawyer would have some previous experience with these kinds of cases and would lean on programming experts for their knowledge. Maybe they are even a former programmer turned lawyer! But I certainly wouldn't hire "a programmer" to represent me no matter how experienced in programming they are. Linus Torvalds would be a great expert witness, but he isn't going to be my general counsel.
      - mieseratte 7y ago
        I would say the lawsuit example isn't great, lawyer is a specialized profession unto itself. Management has it's own vagaries, but I think it's easier for the right kind of engineer to get into management rather than a lifelong manager to gain competence in engineering and that was the original point. Having someone with no background in engineering running a department is suspect.
        txcwpalpha 7y ago
        There's still a disconnect here. Infosec is a specialized profession unto itself, too. A CISO is not just "an engineer that's gone into management". My original point is that security is so far removed from "engineering" that it's incorrect to equate the two (even though it's done all the time). >I think it's easier for the right kind of engineer to get into management rather than a lifelong manager to gain competence in engineering IME, it's the opposite. It depends on your specific goal (are we trying to train someone to be CISO or are we training them to be a SOC team leader?), but it's ridiculously easier (and more effective) to take a person with existing management abilities and teach them about security than it is to take an engineer and teach them security management skills. >Having someone with no background in engineering running a department is suspect. It's really not, because again, engineering != security. It's no more suspect than the CFO not having a background in engineering.
- goatinaboat 7y ago
  Wasn’t that the same at Equifax? https://www.marketwatch.com/story/equifax-ceo-hired-a-music-major-as-the-companys-chief-security-officer-2017-09-15 https://www.marketwatch.com/story/equifax-ceo-hired-a-music-...
- ttul 7y ago
  This is an extreme manipulation of the facts. He has an M.Sc in Computer Science, did R&D at Hewlett-Packard during their golden age, and then spent 22 years at Sandia. This guy was only a bureaucrat in the thinnest interpretation of the word.
- raghava 7y ago
  What according to you is a qualification to be a CISO, of a bank or a financial institution? I would any day put my money in a bank who has a CISO with such credentials as that of Michael Johnson. He has been an engineer, done R&D, does have the right chops too I guess. And CyberSec has multiple layers - mostly the people, which is the weakest one. And that, my friend, cannot really be solved just by tech.
- ilaksh 7y ago
  Whatever he was, if the article is to be believed, it seems like his poor management caused the problem. I believe it. This is often the case, where managers actively interfere with priorities that the engineers try to provide. Not only that but he was using resources to play politics. The issue is that fundamentally that's what managers and executives do. Their profession is politicking and manipulation rather than solving problems.
dangero 7y ago
Not discounting this story, but I’d like to point out that raising concerns is a personal protective strategy for any cyber staff. If you constantly raise concerns that you know cannot or will not be resolved, you have created a paper trail scapegoat for when something goes wrong. The more ambiguous the better: firewall settings not monitored enough, Too much turnover something will fall through cracks, Etc Protecting a large enterprise from cyber attacks is basically impossible, so coping with that stress and protecting your career by looking for other things to blame makes sense to me.
- trabant00 7y ago
  That is an unfair blow. What can you do as an individual in an organization that does not value security and sees it as a sunken cost? I found myself in that position multiple times and I am a sysadmin not security guy. I have no formal responsability in that position and no incentive for covering my ass. But I still have fights with management about it.
  - chii 7y ago
    you don't, except to cover your ass. And so the problem remains unsolved. The law, and regulation, needs to step in. It should be as frowned upon by the law as a restaurant not meeting sanitation requirements.
    - trabant00 7y ago
      > you don't, except to cover your ass. You might want to read my reply once more. > And so the problem remains unsolved. Why do you talk about things you don't know? I've scored multiple victories on that front. > The law, and regulation, needs to step in. I work in PCI environments. The regulations are there but few companies follow them except on paper to cover their ass. Regulations are not the solution. People are. We need to fight.
      - 50656E6973 7y ago
        PCI regulations are weak and do little to cover the majority of threats.
kerng 7y ago
Managment just does basic risk assessment: pay a few million here or there for a fine doesn't really impact business continuity - so security is not important at all. That's the reality. For profit companies are often quite unethical. Laws that put executives in jail if they do not perform basic due diligence might help, e.g. proof a security program is established and executed on, to ensure their own defined standards and policies are met. There are quite a few CSO/CISOs from breached companies who should not be allowed to continue performing their profession. Startups are often perfectly falling into this bucket unfortunately. As soon as one has more then a certain amount of customers, stakes should become extremely high.
- chii 7y ago
  > For profit companies are often quite unethical. this is why i m an advocate of making the law === to ethics. The reason a company is unethical is because those that are ethical are punished by the market (as an unethical, but completely law abiding company can outcompete them by virtue of having less costs involved).
  - billman 7y ago
    I think the problem with this is that there will always be "unethical" ways to work around the legal system. In fact, having more laws can make this easier and more obscure.
  - dec0dedab0de 7y ago
    This is difficult because not everyone believes in the same ethics or follows the same laws. I think it's best to not assume my ethics are somehow more or less valid than anyone else's. That said, I definitely agree that the law needs to change for these situations. It's tricky though, because the entire reason people create corporations is to avoid personal responsibility for their actions.
    - Ascetik 7y ago
      >I think it's best to not assume my ethics are somehow more or less valid than anyone else's. That violates the law of non-contradiction. Either one is wrong and one is right, or both people are wrong and they haven't realized what is right yet. Moral subjectivism as a principle makes absolutely no sense, and Aristotle destroyed that argument in Nichomachean Ethics.
arethuza 7y ago
"9 people 6 months to do" That sounds to me like they had decided that they, for whatever reason, didn't want to do the work.
encoderer 7y ago
well, looks like they caught him. /s What a one-sided hit piece.
Communitivity 7y ago
Hiring great people and getting out of their way is only one of three table legs. The other two are listening when they advise you, and empowering them to effect change.
relaunched 7y ago
This article is devoid of any meaningful analysis and perspective. >The cybersecurity unit—responsible for ensuring Capital One’s firewalls were properly configured and scanning the internet for evidence of a data breach—has cycled through senior leaders and staffers in recent years, according to the people. Firewall team combined with threat-intel? I'm not sure what that has to do with the breach, or preventing it. >Sometimes the broader tech-centric culture of the firm could complicate security, the people said. Technology employees had at times been given free rein to write in many coding languages—so many that it made it harder for the cybersecurity unit to spot problems, according to people familiar with the matter. Super common, but coding languages aren't generally acknowledged as responsible for the hack. If there was a code level vulnerability that most likely would have been caught by static analysis, but wasn't because their scanning tools didn't support a language, this comment would make more sense. >the alleged hacker found that a computer managing communications between the company’s cloud and the public internet was misconfigured—effectively it had weak security settings, the Journal previously reported. Server Side Request Forgery, coupled with overpermissioning, could be explained this way - misconfigured WAF. A poor description of the what we think may be the actual exploit. It goes on and on.
- relaunched 7y ago
  Upon re-read, I don't mentally equate WAF (which is related to the breach, with Firewalls. In my mind, they are two completely different controls. However, I'm not sure if that's a pervasive perspective or just my own. Therefore, my aforementioned statement about firewalls may or may not be valid. YMMV.
  - runamok 7y ago
    ? WAF stands for web application firewall... You don't equate a WAF with a hardware firewall? Just not understanding...
    - relaunched 7y ago
      The reporter said Firewall. When I read that, I don't think of a WAF, rather a more traditional firewall that separates internet from intranet. If others, including the reporter, generally think of WAFs when they read firewall, then my criticism on this particular point is unfair. For example, if someone listed 2 years of experience with firewall rules, I'd automatically think routing and not WAF rules configuration.
aluminussoma 7y ago
I think someone is trying to cover their ass. Think about how this story could have been written: the WSJ reaches out to current and former employees and writes what they say. Those cyber security employees are already feeling the heat and want to preserve their reputation, so they throw someone else under the bus. This was an inside hack by a former AWS employee. It was difficult to protect against. I can't fault Capital One here as much as I can fault AWS. There will always be tension between the Security team and the rest of the company.
- cheeze 7y ago
  > former AWS employee I've yet to see anything that would imply that being an ex employee helped. It seemed that she used a "public" attack vector.
  - ozymandias12 7y ago
    She was a AWS insider. Aware of the common env configurations should have in place and aware that Capital had a few places to poke here and there. I hardly call this a hack. This is more a cake recipe she followed thanks to working directly on AWS and being a crook.
gyanchawdhary 7y ago
We made an interactive demo to show how the hacker exploited the vulnerability https://application.security https://application.security
TheLuddite 7y ago
How about introducing the following law: If an employee is aware of a breach in their company systems and they use the said breach to enrich themselves - they are protected by law against all prosecution related to the said malicious action.
peterwwillis 7y ago
Even if you know that your company is doing nothing to find exposed secrets on the web, can a single engineer just tell their manager, "Hey, you know what? I'm not going to work on this work you assigned me, I'm going to work on this other thing which might be beneficial down the road." Somebody has to groom the "concern" into a story that can be worked on, then assign it to someone. But often this kind of work will get passed over by a manager or team that would rather work on something else, or doesn't see it as important. If you have high turnover, that makes addressing "concerns" all the more difficult, as people aren't around long enough to coordinate working on them. So just "raising concerns" is not going to change anything; somebody at the top has to be listening for them, and somebody in the middle needs to be tracking getting them resolved.
danielecook 7y ago
To this day I still receive emails for another person with my name through cap one 360. Some were regarding overdue bills, and with others some personal information was given. I called repeatedly... and nothing was ever done about it. I think they poorly merged account data at one point - perhaps it’s a broader reflection of their IT work.

141 points by valiant-comma 7y ago

https://www.linkedin.com/in/michael-johnson-098437117/ https://www.linkedin.com/in/michael-johnson-098437117/ They hired a bureaucrat, not an engineer, to be their CISO.

hanniabu 7y ago
Sounds like they hired an Olympian
rficcaglia 7y ago
He worked at Sandia so he seems to have engineering chops. But it is ridiculous to think that one individual or even one isolated group can defend against pervasive threats. Like worker safety, it has to be a cultural priority starting at the top and made priority #1 in every decision and work process. Even then, security is an illusion. Eventually something or someone will find a way to get through. Zero trust needs to be embraced. Assume every employee, partner, developer and vendor is going to have full internal access and abuse it...then defend from there.
txcwpalpha 7y ago
Hiring an engineer (and by that I assume you mean someone who has the bulk of their experience as a software engineer) as CISO is exactly the opposite of what a company should do. Security is absolutely not the same skill set as software engineering, and it's a huge misconception that people equate infosec with programming. Security engineers of course should be programmers, but an enormous amount of work that goes into infosec has absolutely nothing to do with code, and this is especially true for someone high up in management ranks like a CISO where their entire job is dealing with bureaucracy. Hiring an engineer to be a CISO would be like choosing a biologist to give you surgery instead of a surgeon. Yes, a biologist probably knows a lot about the nitty-gritty of how bodies work, but knowing how the body works and knowing how to perform surgery on a body are two very, very different things. That guy's LinkedIn looks more than qualified to be a CISO, and is certainly more qualified than the vast majority of CISOs I worked with in my career as a security consultant. And on top of all that, he has a degree in computer engineering and has prior experience as an engineer, so I have no idea why you're claiming he isn't an engineer.
- goatinaboat 7y ago
  I see where you’re coming from but this is not far off the thinking that “an MBA can manage anything”. To use your analogy no surgeon is allowed to cut until he or she does have a solid grounding in biology.
  - txcwpalpha 7y ago
    You're right, that wasn't a very good analogy. My point is not to say that a CISO/surgeon shouldn't have any experience in engineering/biology (in fact, as you pointed out, it is the opposite! you ideally want a CISO with some technical chops just as you want a surgeon with some biology knowledge). My point is that there is much more to being a CISO than just technical knowledge. Maybe a better analogy would be choosing someone to represent you in a lawsuit about programming patents. You would want a lawyer representing you, not a programmer. Ideally the lawyer would have some previous experience with these kinds of cases and would lean on programming experts for their knowledge. Maybe they are even a former programmer turned lawyer! But I certainly wouldn't hire "a programmer" to represent me no matter how experienced in programming they are. Linus Torvalds would be a great expert witness, but he isn't going to be my general counsel.
    - mieseratte 7y ago
      I would say the lawsuit example isn't great, lawyer is a specialized profession unto itself. Management has it's own vagaries, but I think it's easier for the right kind of engineer to get into management rather than a lifelong manager to gain competence in engineering and that was the original point. Having someone with no background in engineering running a department is suspect.
      - txcwpalpha 7y ago
        There's still a disconnect here. Infosec is a specialized profession unto itself, too. A CISO is not just "an engineer that's gone into management". My original point is that security is so far removed from "engineering" that it's incorrect to equate the two (even though it's done all the time). >I think it's easier for the right kind of engineer to get into management rather than a lifelong manager to gain competence in engineering IME, it's the opposite. It depends on your specific goal (are we trying to train someone to be CISO or are we training them to be a SOC team leader?), but it's ridiculously easier (and more effective) to take a person with existing management abilities and teach them about security than it is to take an engineer and teach them security management skills. >Having someone with no background in engineering running a department is suspect. It's really not, because again, engineering != security. It's no more suspect than the CFO not having a background in engineering.
goatinaboat 7y ago
Wasn’t that the same at Equifax? https://www.marketwatch.com/story/equifax-ceo-hired-a-music-major-as-the-companys-chief-security-officer-2017-09-15 https://www.marketwatch.com/story/equifax-ceo-hired-a-music-...
ttul 7y ago
This is an extreme manipulation of the facts. He has an M.Sc in Computer Science, did R&D at Hewlett-Packard during their golden age, and then spent 22 years at Sandia. This guy was only a bureaucrat in the thinnest interpretation of the word.
raghava 7y ago
What according to you is a qualification to be a CISO, of a bank or a financial institution? I would any day put my money in a bank who has a CISO with such credentials as that of Michael Johnson. He has been an engineer, done R&D, does have the right chops too I guess. And CyberSec has multiple layers - mostly the people, which is the weakest one. And that, my friend, cannot really be solved just by tech.
ilaksh 7y ago
Whatever he was, if the article is to be believed, it seems like his poor management caused the problem. I believe it. This is often the case, where managers actively interfere with priorities that the engineers try to provide. Not only that but he was using resources to play politics. The issue is that fundamentally that's what managers and executives do. Their profession is politicking and manipulation rather than solving problems.

dangero 7y ago

Not discounting this story, but I’d like to point out that raising concerns is a personal protective strategy for any cyber staff. If you constantly raise concerns that you know cannot or will not be resolved, you have created a paper trail scapegoat for when something goes wrong. The more ambiguous the better: firewall settings not monitored enough, Too much turnover something will fall through cracks, Etc Protecting a large enterprise from cyber attacks is basically impossible, so coping with that stress and protecting your career by looking for other things to blame makes sense to me.

trabant00 7y ago
That is an unfair blow. What can you do as an individual in an organization that does not value security and sees it as a sunken cost? I found myself in that position multiple times and I am a sysadmin not security guy. I have no formal responsability in that position and no incentive for covering my ass. But I still have fights with management about it.
- chii 7y ago
  you don't, except to cover your ass. And so the problem remains unsolved. The law, and regulation, needs to step in. It should be as frowned upon by the law as a restaurant not meeting sanitation requirements.
  - trabant00 7y ago
    > you don't, except to cover your ass. You might want to read my reply once more. > And so the problem remains unsolved. Why do you talk about things you don't know? I've scored multiple victories on that front. > The law, and regulation, needs to step in. I work in PCI environments. The regulations are there but few companies follow them except on paper to cover their ass. Regulations are not the solution. People are. We need to fight.
    - 50656E6973 7y ago
      PCI regulations are weak and do little to cover the majority of threats.

kerng 7y ago

Managment just does basic risk assessment: pay a few million here or there for a fine doesn't really impact business continuity - so security is not important at all. That's the reality. For profit companies are often quite unethical. Laws that put executives in jail if they do not perform basic due diligence might help, e.g. proof a security program is established and executed on, to ensure their own defined standards and policies are met. There are quite a few CSO/CISOs from breached companies who should not be allowed to continue performing their profession. Startups are often perfectly falling into this bucket unfortunately. As soon as one has more then a certain amount of customers, stakes should become extremely high.

chii 7y ago
> For profit companies are often quite unethical. this is why i m an advocate of making the law === to ethics. The reason a company is unethical is because those that are ethical are punished by the market (as an unethical, but completely law abiding company can outcompete them by virtue of having less costs involved).
- billman 7y ago
  I think the problem with this is that there will always be "unethical" ways to work around the legal system. In fact, having more laws can make this easier and more obscure.
- dec0dedab0de 7y ago
  This is difficult because not everyone believes in the same ethics or follows the same laws. I think it's best to not assume my ethics are somehow more or less valid than anyone else's. That said, I definitely agree that the law needs to change for these situations. It's tricky though, because the entire reason people create corporations is to avoid personal responsibility for their actions.
  - Ascetik 7y ago
    >I think it's best to not assume my ethics are somehow more or less valid than anyone else's. That violates the law of non-contradiction. Either one is wrong and one is right, or both people are wrong and they haven't realized what is right yet. Moral subjectivism as a principle makes absolutely no sense, and Aristotle destroyed that argument in Nichomachean Ethics.

arethuza 7y ago

"9 people 6 months to do" That sounds to me like they had decided that they, for whatever reason, didn't want to do the work.

encoderer 7y ago

well, looks like they caught him. /s What a one-sided hit piece.

Communitivity 7y ago

Hiring great people and getting out of their way is only one of three table legs. The other two are listening when they advise you, and empowering them to effect change.

relaunched 7y ago

This article is devoid of any meaningful analysis and perspective. >The cybersecurity unit—responsible for ensuring Capital One’s firewalls were properly configured and scanning the internet for evidence of a data breach—has cycled through senior leaders and staffers in recent years, according to the people. Firewall team combined with threat-intel? I'm not sure what that has to do with the breach, or preventing it. >Sometimes the broader tech-centric culture of the firm could complicate security, the people said. Technology employees had at times been given free rein to write in many coding languages—so many that it made it harder for the cybersecurity unit to spot problems, according to people familiar with the matter. Super common, but coding languages aren't generally acknowledged as responsible for the hack. If there was a code level vulnerability that most likely would have been caught by static analysis, but wasn't because their scanning tools didn't support a language, this comment would make more sense. >the alleged hacker found that a computer managing communications between the company’s cloud and the public internet was misconfigured—effectively it had weak security settings, the Journal previously reported. Server Side Request Forgery, coupled with overpermissioning, could be explained this way - misconfigured WAF. A poor description of the what we think may be the actual exploit. It goes on and on.

relaunched 7y ago
Upon re-read, I don't mentally equate WAF (which is related to the breach, with Firewalls. In my mind, they are two completely different controls. However, I'm not sure if that's a pervasive perspective or just my own. Therefore, my aforementioned statement about firewalls may or may not be valid. YMMV.
- runamok 7y ago
  ? WAF stands for web application firewall... You don't equate a WAF with a hardware firewall? Just not understanding...
  - relaunched 7y ago
    The reporter said Firewall. When I read that, I don't think of a WAF, rather a more traditional firewall that separates internet from intranet. If others, including the reporter, generally think of WAFs when they read firewall, then my criticism on this particular point is unfair. For example, if someone listed 2 years of experience with firewall rules, I'd automatically think routing and not WAF rules configuration.

aluminussoma 7y ago

I think someone is trying to cover their ass. Think about how this story could have been written: the WSJ reaches out to current and former employees and writes what they say. Those cyber security employees are already feeling the heat and want to preserve their reputation, so they throw someone else under the bus. This was an inside hack by a former AWS employee. It was difficult to protect against. I can't fault Capital One here as much as I can fault AWS. There will always be tension between the Security team and the rest of the company.

cheeze 7y ago
> former AWS employee I've yet to see anything that would imply that being an ex employee helped. It seemed that she used a "public" attack vector.
- ozymandias12 7y ago
  She was a AWS insider. Aware of the common env configurations should have in place and aware that Capital had a few places to poke here and there. I hardly call this a hack. This is more a cake recipe she followed thanks to working directly on AWS and being a crook.

gyanchawdhary 7y ago

We made an interactive demo to show how the hacker exploited the vulnerability https://application.security https://application.security

TheLuddite 7y ago

How about introducing the following law: If an employee is aware of a breach in their company systems and they use the said breach to enrich themselves - they are protected by law against all prosecution related to the said malicious action.

peterwwillis 7y ago

Even if you know that your company is doing nothing to find exposed secrets on the web, can a single engineer just tell their manager, "Hey, you know what? I'm not going to work on this work you assigned me, I'm going to work on this other thing which might be beneficial down the road." Somebody has to groom the "concern" into a story that can be worked on, then assign it to someone. But often this kind of work will get passed over by a manager or team that would rather work on something else, or doesn't see it as important. If you have high turnover, that makes addressing "concerns" all the more difficult, as people aren't around long enough to coordinate working on them. So just "raising concerns" is not going to change anything; somebody at the top has to be listening for them, and somebody in the middle needs to be tracking getting them resolved.

danielecook 7y ago

To this day I still receive emails for another person with my name through cap one 360. Some were regarding overdue bills, and with others some personal information was given. I called repeatedly... and nothing was ever done about it. I think they poorly merged account data at one point - perhaps it’s a broader reflection of their IT work.