8 ms·
I work at a government agency and here are my tales. 1) They install a root certificate on all machines and use that to MITM all TLS connections using a firewa
by huy-nguyen 4y ago
I work at a government agency and here are my tales.
1) They install a root certificate on all machines and use that to MITM all TLS connections using a firewall appliance. They turn this MITM on one day without notifying any developer. Overnight, all our builds (run on-prem) fail because npm install, pip install etc fail and we spent a long time trying to figure it out. They are still failing to this day and I have to get off the VPN every time I need to run these simple commands. IT absolutely doesn't give a flying **** about developers.
2) They ban all non-Chrome browsers from being installed. As in, if you install such a browser and try to launch it, the system will say "browser X is banned. Contact IT." They would have banned Safari too had it not been part of the OS. Furthermore, they also disabled private browsing in Chrome (probably the ability to do this is why they allow Chrome). I think they're preventing people from hiding their internet browsing.
- bombcar 4y agoThat first one indicates something is being injected and the checksums are failing, that's ... worrying. Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess.
- Karunamon 4y agoI would be willing to bet the first one is caused more by those tools not being aware of the firewall appliance CA rather than failing checksums. Doing your own certificates at scale is a pain in the ass because every tool/container has its own way of handling the trusted list.
- thayne 4y ago> Or npm and pip use their own certificate stacks and refuse the firewall's cert, which is ... good I guess. Combined with the fact that chrome is the only allowed browser, I suspect it is the other way around. Chrome uses its own certificate stack, and I would guess IT only added the MITM certificate to the chrome trusted CA list, not the system one.
- marcosdumay 4y agoIt could be just the problem of the certificate being invalid for those tools because the MITM one was installed only for Chrome up to the firewall replacing all of the files with html pages of some antivirus with internal links where the user can download them. Corporate middleboxes come in all shades of stupid.
- thaeli 4y agoWhy are they allowing you to run npm, pip, etc from public repositories at all? That's a huge supply chain risk. If builds are worth doing on prem they also need to be pulling solely from internal, vetted repositories.
- SSLy 4y agoBold of you to assume the developers were given a budget and staff to run on-prem repo mirror.
- dspillett 4y ago> Why are they allowing you to Maybe “you shouldn't be doing that anyway” is a key part of why they don't care to spend effort resolving the problem.
- flumpcakes 4y agoThis is a big worry of mine. I've worked in various places from big finance, public sector, to privately owned. It was only the big finance institution (the biggest) that actually seemed to care about supply chain attacks. Everything was locked down super well and you could not! use a random library without it getting vetted by a central team. In fact, we were even locked down to specific versions of programming languages. People see this as annoying and in the way of developers, but it really is the only way to secure your development "supply chain". When people cry about this I always ask: do you really want the entire financial industry grinding to a halt because someone took down left-pad?
- TheRealDunkirk 4y agoThe idea that some team is "vetting" that the entire stack of stuff you'd pull from npm for a React front-end app is "safe" is ridiculous. Forget the mirroring; that's trivial. What criteria or process would make you think you had a "vetted" snapshot, beyond what they already do!?
- slt2021 4y agoThis process is automated by static code analysis tools, once it is deployed then absolutely no meatbag effort is required
- jjoonathan 4y agoThe #1 value proposition of the cloud is escaping dogshit expense processes and the #2 value proposition of the cloud is escaping dogshit IT.
- raxxorraxor 4y ago> They ban all non-Chrome browsers from being installed. There are portable apps for other browsers, Firefox for example. FF has its own certificate store that overwhelms IT. about:config -> security.enterprise_roots.enabled and it uses the system store. Overall Firefox is a very good browser to configure for different machines. The out of the box Chrome or Edge are just a bit more forgiving to MITM attacks. So the user doesn't notice perhaps? Aside from that they are horrible browsers with horrible priorities.
- dspillett 4y ago> IT absolutely doesn't give a flying ** about developers. They are not paid to. Their performance is judged against how close they get to zero compliance issues, not how close they get to zero times developers were unhappy! > I think they're preventing people from hiding their internet browsing. Without delving into the “do you have the right to privacy even on a company machine”, who would be daft enough to do something they want to hide from the company on a company machine, or the company network at all? Though there are valid useful uses of pron^H^H^H^Hicognito mode so it seems silly to ban it.
- wahnfrieden 4y agoYou’re just arguing for surveillance (by capital or state) with the tired line of you should have nothing to worry about unless you deserve it which is absurd/reactionary
- dspillett 4y agoI'm not saying it is right for people to be monitored, but that I would never trust that I wasn't being so I'd not be daft enough to do something I don't want the company to know about using their resources. And there are perfectly valid reasons for companies to monitor traffic: data exfiltration, accidental or malicious, is a significant concern for companies that hold and process PII and for the people who have their PII held/processed by those companies. It is not as black & white as “monitoring and surveillance bad” unless you only care about your personal privacy.
- sillystuff 4y agoSome of these "security" products that MiM TLS traffic allow configurations that objectively reduce your security. You can configure Palo Alto devices to accept a self-signed cert from the Internet, but present your trusted MiM cert to the on-site user. Now the user isn't aware that they are the victim of a second MiM outside the organization. The organization also exposes itself to greater liability. E.g., a rogue employee could use the trusted MiM CA cert for their own MiM e.g., capturing banking credentials of co-workers or accessing user/employee PII they would otherwise not have access to. Yes, monitoring traffic by MiM https to external sites can alert you to / possibly prevent accidental exfiltration, but it doesn't prevent intentional exfiltration. It is, however, very effective at monitoring employees. The thing it is best at, might be its true purpose in an organization.
- deepsun 4y agoWell, if we're talking about security, their ban on NPM is a good thing, that's a huge supply chain risk. If you don't have a budget for the vetted repositories, it means you don't have a budget for the project within the security requirements. You shouldn't be circumventing the security requirements, you should escalate the issue. PS: of course I'm not talking about other things like MITM certs, that only reduces security.
- xani_ 4y agoAnd that kids is how it looks when security team just sits in their ivory tower and shits on everyone else in name of security theathre they're paid to play > Overnight, all our builds (run on-prem) fail because npm install, pip install etc fail and we spent a long time trying to figure it out. They are still failing to this day and I have to get off the VPN every time I need to run these simple commands. IT absolutely doesn't give a flying ** about developers. Add their cert to system store ? Won't help inside containers tho... without much fuckery
- bornfreddy 4y agoWhy not? Just mount a volume with certs and you're good to go. In every container, of course.
- richbell 4y ago> Add their cert to system store? That's the fun part: every technology and tool has its own bespoke way of handling certificates, and it often isn't as simple as adding a certificate to the system store.
- slt2021 4y agoYou are not supposed to mess with certificates inside containers. Security team should have provided you with golden image with hardened config, latest patches installed, and corporate certs installed in certificate store. If they didnt, they aint doing correct DevSecOps/SecDevOps or whatever the fancy term is for integrating security within development team. It is a big red flag that any developer can pull whatever image for container running in production, possibly with unpatched vulnerabilities and loose config and ports open, and running with root privileges, etc. Usually stuff has to be vetted and checked prior to being deployed in production environment
- Spooky23 4y agoFor good reason. Stuff like that is a really high risk and won’t meet audit standards. I’m in charge of the IT goons somewhere. We aspire to provide a better level of service and maintain local repos of things you’re allowed to use. Stuff like Node isn’t allowed near anything important though. I would be careful. An agency doing stuff like that is probably running an EDR that will detect and report on what you’re doing. If it catches what you’re up to, you’ll be jammed up.
- _fat_santa 4y agoIt might very well be for a good reason. But in my experience, it's never the policies but the communication. IT was right to make whatever policy change they needed to, the fucked up by not telling any of the dev teams.
- Spooky23 4y ago100% agree. Most fubar things are caused by poor communication, lack of empathy and poor understanding.
- sillystuff 4y agoThe MiM might not be your IT folks, but rather management. I was in a meeting which included folks from Palo Alto (PA) and management where PA was hard selling their ability to MiM all https connections and link all activities of the users to their usernames through various methods from directory integration to log scraping on radius servers. The managers were super excited about the possibilities. Management not only wanted to implement this, but wanted to do so in secret. IT folks were pushing back-- hard. Firewall as bossware. Firefox being banned is because it uses its own certificate store, so Firefox users would see a browser warning every time they visit any https site notifying them that their traffic is being MiM'd. Chrome and chrome reskins like MS Edge use the OS store which MS Windows centric organizations can easily (centrally using MS tools [GP]) add the trusted CA for MiM into. For the Macs, it probably wouldn't matter since the 3rd party mgmt tools could probably push out either.
- justsomehnguy 4y ago> Firefox being banned is because it uses its own certificate store FYI You can instruct FF to use system trust store: https://support.mozilla.org/en-US/kb/setting-certificate-authorities-firefox https://support.mozilla.org/en-US/kb/setting-certificate-aut...
- throwaway743 4y agoCaught an ex employer using sslstrip and they definitely used bossware. Management would imply they were reading work and personal messages, emails, browsing through thinly veiled threats to workers (self included) and through gossip. They also used push notifications on the desktops to know when people were active or what they were doing, and had keyloggers installed/active. Once caught a manager's personal laptop on the network running mitm software. A friendly coworker in IT confirmed all of this with me in private. Tried warning a couple coworkers, but got brushed off. People don't seem to care nor believe even though they're being manipulated. That place was a nightmare to say the least
- TheRealDunkirk 4y agoOh, how I have learned the hard way on this. Our IT now blocks outbound SSH entirely. You know, the secure way to access VM's in, say, our cloud? Sigh. I'm sure there's a "jump" server somewhere that I'd have to log into, `sudo` to another account, THEN SSH to my target box. Whatever. I just avoid the VPN. I used to use `cntlm` to tunnel requests through our firewall for things like Ruby's bundler, as it required NTLM authentication. Now they've also gone the additional mile, and installed a certificate (Cisco Umbrella) in all of our computers, and require its signature to pass the firewall. Unfortunately, it took me a long time to sort this out: why `cntlm` no longer worked, and why none of the usual suggestions on SO fixed it. I finally figured out that RubyInstaller for Windows included a nice facility to deal with this. You just place additional certs in a directory, run a Ruby script, and it will bundle the whole stack into a single .pem, which it will reference for all network-related commands. Thankfully, bundler's error messages were telling me the specific certs I needed, and I could download them from Cisco's web site. Just about a month ago, my company started requiring that cert for ALL traffic, not just HTTP(S). Like for, say, Postgres connections on port 5432. I finally realized that I could reference that same SSL bundle in my Postgres client connections, and get through. I've spent about 8 years here now, and it's been a cat-and-mouse game the whole time. I'm always wondering what's coming next.
- slt2021 4y agoyour practices are the epitome of Shadow IT that company management doesnt like and fights
- TheRealDunkirk 4y agoYou don't know the half of it.
- xist 4y agoThe way you brush it off is insane. Using a jumpbox IS more secure. I understand this may cause problems for your workflow (though there are many ways to work within the confines) it sounds like you're stubbornly insisting your way is the best (and therefore most secure) way. This reeks of entitlement. Work with people and stop being a prima donna, you're not above security concerns.
- slt2021 4y agothis is because "pip install" is insecure, this is supply chain risk. Your IT team should have provided local artifactory proxy through which you can pip install. you should use this command "pip install -i http://artifactory.mycompany.local http://artifactory.mycompany.local pandas" and get url for artifactory from admins
- TheRealDunkirk 4y agoIf "your IT team" has merely created a snapshot of an external repo, how is this any more "secure?" I've asked a similar question below. I really want to understand the thinking here. No IT department is going to go line-by-line through all the packages in "artificatory" or Ruby gems or NPM packages or NuGet's repo, checking them all against known vulnerabilities. No one's going to vet the actual code. If there's a public advisory for one of the packages, the parent repo is going to fix it first, and the internal repo may already be compromised, and the "IT team" is going to have to duplicate the work that the repo runners are already doing, and do it slower. I'm lost here.
- slt2021 4y agothere are IT security vendors that provide static code analysis and scanning for known signatures, that can detect and block malicious packages. Just target SCA at local artifactory and this will be a solved problem. CISO just needs to buy solution and IT admins just needs to deploy that software once and it will keep scanning. Absolutely no extra work from meatbags is required
- TheRealDunkirk 4y agoSo these bulk scanners exist, and the issue is a solved problem, but none of the "root" repos for the popular language stacks are using them? It seems that Microsoft has built an internal tool that runs such a scan on NuGet (https://devblogs.microsoft.com/nuget/how-to-scan-nuget-packages-for-security-vulnerabilities/ https://devblogs.microsoft.com/nuget/how-to-scan-nuget-packa...), at least against your individual app's packages. (That would be a very rare h/t to Microsoft from me.) EDIT: Apparently, you can also do this with npm packages (https://docs.npmjs.com/auditing-package-dependencies-for-security-vulnerabilities https://docs.npmjs.com/auditing-package-dependencies-for-sec...). I don't see any facility to do this with Ruby gems. It looks like the common practice would be to outsource the issue database to GitHub, and let whatever scanner you're using cross-reference that list? What happens when it finds a reported problem? Does it automatically delete that mirrored package, and/or block it from being downloaded or used from the on-prem repo? This is all new to me, and has helped put this in context, but what actual software are you talking about using for analysis? EDIT EDIT: Running `yarn audit` in my main Rails app (just using webpacker to bundle the JS): 97 vulnerabilities found - Packages audited: 1074 Severity: 2 Low | 34 Moderate | 52 High | 9 Critical I just did a `yarn upgrade` about a week ago, so it's not like I'm completely out of date. What would a centrally-managed SCA do about this situation?
- ornornor 4y agoWorked at an insane place that did 1. “It’s more secure” they said. The “solution”? Disable certificate checking of course! What could go wrong? Same place that ran a vulnerable instance of nexus (the package manager) for all internal npm and maven packages for a whole year before patching. It was publicly accessible. And it had a banner on the homepage that said “this version is vulnerable (severity 10/10 anonymous RCE), update NOW”. Anyone who went to https://nexus.1337company.com https://nexus.1337company.com would see it. That company did software for the government. I’m sure I wasn’t the only one who noticed the vulnerability and that some packages got tainted. But we’ll never know because no audit was ever performed and there were no backups of that server anyway. Like I said, absolute joke of a workplace.