4 ms·

Company I worked at turned on MITM TLS for everything. Suddenly a lot of stuff stopped working, because not every piece of software on my machine uses the OS's

Company I worked at turned on MITM TLS for everything. Suddenly a lot of stuff stopped working, because not every piece of software on my machine uses the OS's certificate store. For example Docker containers who then curl to set up stuff.

unethical_ban 2y ago
Yeah, it needs to be rolled out and tested a lot. And not all traffic should be decrypted, namely banking and health.
sebazzz 2y ago
I hate ZScaler with a passion.
- Sohcahtoa82 2y ago
  At my last job, my CISO made the company trial ZScaler. When it caused tons of problems for Engineering, he cancelled the trial, then tried out CloudFlare's MitM proxy, which of course had the same problems. I had to talk to him and say look, what you're trying to do isn't possible without breaking things. It's not a limitation of the products, it's a limitation of the underlying security that you're trying to bypass, and you're making security WORSE, not BETTER. He wanted to log every employee's Internet traffic so that he could determine if someone was leaking company data, such as source code, customer data, etc. I said that there's nothing you can do to stop that, and even if someone knows their traffic is being monitored, it's easy to bypass by just adding a second layer of encryption.
neilv 2y ago
One company had a related (but less defensible) decision, coming down from the top, which broke CI runners and other things, and the poor overworked Git&CI infra lead was trying to work around it. They asked me to be a Git reviewer for their big workaround, and I found around a dozen new vulnerabilities and future build-breaking defects that the workaround introduced. I also told them that it's unreasonable for this other team to ask them to work around that mess, and the solution is for the other team to do the thing in a different, much simpler way. I also ended up raising some even bigger security decision problems with the C-suite. As one engineer from a different team told me (after they'd given notice of leaving, and kindly granted me an exit interview), the company had too many people making other people's jobs harder, for no reason. A too-entrenched-to-fail-soon company might be able to justify that (say, it was the easiest way to check off a compliance box, with their encumbered velocity at implementing anything). Most companies aren't that, though.
- xyzzy123 2y ago
  The root causes are usually among: Security teams are often staffed by people who have no operational experience, and do not understand the consequences of what they are recommending or even mandating. Often those staff are blindly following hardening guides or asking for every configuration switch to be flipped to "most secure" setting without having a good understanding of the threat model for the workload and without taking into account the tradeoffs between utility and operational cost. The level of advice can be on par with ChatGPT or worse, but it is taken more seriously due to the advice-giver's job title. Security teams often have no "skin in the game". There are no real disincentives to stop them from asking for crazy or very expensive things and imposing high costs on other teams. In fact they are incentivised to do that very thing, because the only thing that covers your butt more than recommending everything possible, is recommending everything possible PLUS some things that can't be done with the time & budget available, leaving them able to say "we see you had $security_problem, well, we recommended $impossible_thing but you didn't do it" (e.g. say, $500k of DLP solution [with its own operational risks!] for a workload that only makes $1M a year). To be fair I've seen some good & practical security teams but once you get a bad actor / games player at management level that behaviour can become very sticky. I have seen variants of this nearly everywhere I've worked, it seems very hard to get incentives aligned between the do-ers and the secure-ers. The most practical workaround I've seen is to make sure there is a reasonable balance of political power between the various parties. "Reasonable" can be hard to establish but is context dependent. You would expect "Security" to have more power in an F500 because the brand value, financial and legal exposure are high and individual dev teams aren't necessarily across or exposed to the full consequences of the damage they can cause. In a startup you would expect "Security" to have much less power because the consequences of not shipping / misallocating effort are almost immediately existential.
  - bostik 2y ago
    You nailed it. In my experience these "security" teams accumulate people who want PM level powers without having to deal with any of the accountability. The fact is that security is a holistic concept and can ONLY be evaluated in context. That in turn requires good technical and operational knowledge. But people like that are rare and expensive, so instead we have entire armies of box-tickers who lack the intellectual capacity to even understand what a "tradeoff" means. Something went terribly wrong around the mid-90s, when security changed from a discipline practised and understood by hands-on professionals into a consulting gig. Disclosure: I've been doing infosec professionally since 1993.

unethical_ban 2y ago

Yeah, it needs to be rolled out and tested a lot. And not all traffic should be decrypted, namely banking and health.

sebazzz 2y ago

I hate ZScaler with a passion.

Sohcahtoa82 2y ago
At my last job, my CISO made the company trial ZScaler. When it caused tons of problems for Engineering, he cancelled the trial, then tried out CloudFlare's MitM proxy, which of course had the same problems. I had to talk to him and say look, what you're trying to do isn't possible without breaking things. It's not a limitation of the products, it's a limitation of the underlying security that you're trying to bypass, and you're making security WORSE, not BETTER. He wanted to log every employee's Internet traffic so that he could determine if someone was leaking company data, such as source code, customer data, etc. I said that there's nothing you can do to stop that, and even if someone knows their traffic is being monitored, it's easy to bypass by just adding a second layer of encryption.

neilv 2y ago

One company had a related (but less defensible) decision, coming down from the top, which broke CI runners and other things, and the poor overworked Git&CI infra lead was trying to work around it. They asked me to be a Git reviewer for their big workaround, and I found around a dozen new vulnerabilities and future build-breaking defects that the workaround introduced. I also told them that it's unreasonable for this other team to ask them to work around that mess, and the solution is for the other team to do the thing in a different, much simpler way. I also ended up raising some even bigger security decision problems with the C-suite. As one engineer from a different team told me (after they'd given notice of leaving, and kindly granted me an exit interview), the company had too many people making other people's jobs harder, for no reason. A too-entrenched-to-fail-soon company might be able to justify that (say, it was the easiest way to check off a compliance box, with their encumbered velocity at implementing anything). Most companies aren't that, though.

xyzzy123 2y ago
The root causes are usually among: Security teams are often staffed by people who have no operational experience, and do not understand the consequences of what they are recommending or even mandating. Often those staff are blindly following hardening guides or asking for every configuration switch to be flipped to "most secure" setting without having a good understanding of the threat model for the workload and without taking into account the tradeoffs between utility and operational cost. The level of advice can be on par with ChatGPT or worse, but it is taken more seriously due to the advice-giver's job title. Security teams often have no "skin in the game". There are no real disincentives to stop them from asking for crazy or very expensive things and imposing high costs on other teams. In fact they are incentivised to do that very thing, because the only thing that covers your butt more than recommending everything possible, is recommending everything possible PLUS some things that can't be done with the time & budget available, leaving them able to say "we see you had $security_problem, well, we recommended $impossible_thing but you didn't do it" (e.g. say, $500k of DLP solution [with its own operational risks!] for a workload that only makes $1M a year). To be fair I've seen some good & practical security teams but once you get a bad actor / games player at management level that behaviour can become very sticky. I have seen variants of this nearly everywhere I've worked, it seems very hard to get incentives aligned between the do-ers and the secure-ers. The most practical workaround I've seen is to make sure there is a reasonable balance of political power between the various parties. "Reasonable" can be hard to establish but is context dependent. You would expect "Security" to have more power in an F500 because the brand value, financial and legal exposure are high and individual dev teams aren't necessarily across or exposed to the full consequences of the damage they can cause. In a startup you would expect "Security" to have much less power because the consequences of not shipping / misallocating effort are almost immediately existential.
- bostik 2y ago
  You nailed it. In my experience these "security" teams accumulate people who want PM level powers without having to deal with any of the accountability. The fact is that security is a holistic concept and can ONLY be evaluated in context. That in turn requires good technical and operational knowledge. But people like that are rare and expensive, so instead we have entire armies of box-tickers who lack the intellectual capacity to even understand what a "tradeoff" means. Something went terribly wrong around the mid-90s, when security changed from a discipline practised and understood by hands-on professionals into a consulting gig. Disclosure: I've been doing infosec professionally since 1993.