4 ms·

One company had a related (but less defensible) decision, coming down from the top, which broke CI runners and other things, and the poor overworked Git&CI infr

by neilv 3y ago

One company had a related (but less defensible) decision, coming down from the top, which broke CI runners and other things, and the poor overworked Git&CI infra lead was trying to work around it. They asked me to be a Git reviewer for their big workaround, and I found around a dozen new vulnerabilities and future build-breaking defects that the workaround introduced. I also told them that it's unreasonable for this other team to ask them to work around that mess, and the solution is for the other team to do the thing in a different, much simpler way. I also ended up raising some even bigger security decision problems with the C-suite. As one engineer from a different team told me (after they'd given notice of leaving, and kindly granted me an exit interview), the company had too many people making other people's jobs harder, for no reason. A too-entrenched-to-fail-soon company might be able to justify that (say, it was the easiest way to check off a compliance box, with their encumbered velocity at implementing anything). Most companies aren't that, though.

xyzzy123 3y ago
The root causes are usually among: Security teams are often staffed by people who have no operational experience, and do not understand the consequences of what they are recommending or even mandating. Often those staff are blindly following hardening guides or asking for every configuration switch to be flipped to "most secure" setting without having a good understanding of the threat model for the workload and without taking into account the tradeoffs between utility and operational cost. The level of advice can be on par with ChatGPT or worse, but it is taken more seriously due to the advice-giver's job title. Security teams often have no "skin in the game". There are no real disincentives to stop them from asking for crazy or very expensive things and imposing high costs on other teams. In fact they are incentivised to do that very thing, because the only thing that covers your butt more than recommending everything possible, is recommending everything possible PLUS some things that can't be done with the time & budget available, leaving them able to say "we see you had $security_problem, well, we recommended $impossible_thing but you didn't do it" (e.g. say, $500k of DLP solution [with its own operational risks!] for a workload that only makes $1M a year). To be fair I've seen some good & practical security teams but once you get a bad actor / games player at management level that behaviour can become very sticky. I have seen variants of this nearly everywhere I've worked, it seems very hard to get incentives aligned between the do-ers and the secure-ers. The most practical workaround I've seen is to make sure there is a reasonable balance of political power between the various parties. "Reasonable" can be hard to establish but is context dependent. You would expect "Security" to have more power in an F500 because the brand value, financial and legal exposure are high and individual dev teams aren't necessarily across or exposed to the full consequences of the damage they can cause. In a startup you would expect "Security" to have much less power because the consequences of not shipping / misallocating effort are almost immediately existential.
- bostik 3y ago
  You nailed it. In my experience these "security" teams accumulate people who want PM level powers without having to deal with any of the accountability. The fact is that security is a holistic concept and can ONLY be evaluated in context. That in turn requires good technical and operational knowledge. But people like that are rare and expensive, so instead we have entire armies of box-tickers who lack the intellectual capacity to even understand what a "tradeoff" means. Something went terribly wrong around the mid-90s, when security changed from a discipline practised and understood by hands-on professionals into a consulting gig. Disclosure: I've been doing infosec professionally since 1993.
cduzz 3y ago
I get that this sort of MITM often breaks CI pipes, but working with this shouldn't be anything more than a chunk of work involving stuff you ought to be doing already, including: 1. put all assets into infrastructure you control, allow pulling only from that infrastructure, no exceptions. 2. understand your containers enough to be able to modify and manage their trusted CA / TLS chains. You really ought to be in a position to manage your supply chain and you really ought to be in a position to manage what your containers trust. Certainly more work and inconvenient, and maybe not practical everywhere, but worth doing even independently of a middleware box, at least once you get to some level of "in production for reals"
- neilv 3y ago
  Agreed, and that wasn't the problem.

by neilv 3y ago

xyzzy123 3y ago

The root causes are usually among: Security teams are often staffed by people who have no operational experience, and do not understand the consequences of what they are recommending or even mandating. Often those staff are blindly following hardening guides or asking for every configuration switch to be flipped to "most secure" setting without having a good understanding of the threat model for the workload and without taking into account the tradeoffs between utility and operational cost. The level of advice can be on par with ChatGPT or worse, but it is taken more seriously due to the advice-giver's job title. Security teams often have no "skin in the game". There are no real disincentives to stop them from asking for crazy or very expensive things and imposing high costs on other teams. In fact they are incentivised to do that very thing, because the only thing that covers your butt more than recommending everything possible, is recommending everything possible PLUS some things that can't be done with the time & budget available, leaving them able to say "we see you had $security_problem, well, we recommended $impossible_thing but you didn't do it" (e.g. say, $500k of DLP solution [with its own operational risks!] for a workload that only makes $1M a year). To be fair I've seen some good & practical security teams but once you get a bad actor / games player at management level that behaviour can become very sticky. I have seen variants of this nearly everywhere I've worked, it seems very hard to get incentives aligned between the do-ers and the secure-ers. The most practical workaround I've seen is to make sure there is a reasonable balance of political power between the various parties. "Reasonable" can be hard to establish but is context dependent. You would expect "Security" to have more power in an F500 because the brand value, financial and legal exposure are high and individual dev teams aren't necessarily across or exposed to the full consequences of the damage they can cause. In a startup you would expect "Security" to have much less power because the consequences of not shipping / misallocating effort are almost immediately existential.

bostik 3y ago
You nailed it. In my experience these "security" teams accumulate people who want PM level powers without having to deal with any of the accountability. The fact is that security is a holistic concept and can ONLY be evaluated in context. That in turn requires good technical and operational knowledge. But people like that are rare and expensive, so instead we have entire armies of box-tickers who lack the intellectual capacity to even understand what a "tradeoff" means. Something went terribly wrong around the mid-90s, when security changed from a discipline practised and understood by hands-on professionals into a consulting gig. Disclosure: I've been doing infosec professionally since 1993.

cduzz 3y ago

I get that this sort of MITM often breaks CI pipes, but working with this shouldn't be anything more than a chunk of work involving stuff you ought to be doing already, including: 1. put all assets into infrastructure you control, allow pulling only from that infrastructure, no exceptions. 2. understand your containers enough to be able to modify and manage their trusted CA / TLS chains. You really ought to be in a position to manage your supply chain and you really ought to be in a position to manage what your containers trust. Certainly more work and inconvenient, and maybe not practical everywhere, but worth doing even independently of a middleware box, at least once you get to some level of "in production for reals"

neilv 3y ago
Agreed, and that wasn't the problem.