8 ms·

Stop Breaking TLS

175 points by todsacerdoti 11mo ago

hacker_homie 11mo ago
But I need to see what they are googling! /sarcasm
MathMonkeyMan 11mo ago
I remember at my first job, the internet stopped working at my workstation. I got on the phone with IT, and the guy said "looks like you don't have our new certificates." I asked why I would need my employer's certificates. He said "because we MITM every connection." I asked if that was even legal, and he said yes it's legal. At another job I was handling a support ticket where a customer was asking, in so many words, "can I get HTTP headers of requests flowing through my Envoy TLS reverse proxy?" I said that they could terminate TLS at the proxy and redo things that way, but then that wouldn't be a TLS proxy it'd be a MITM or a gateway. They could log the downstream/upstream and duration of connections, but that wouldn't help.
- cxr 11mo ago
  No one who understands what "MITM" means should have any expectation that I/O with a device owned and administered by a third party can be trusted (whether they do it by subverting PKI with internal certificate or not).
phoronixrly 11mo ago
Hey, allowing your employees to have secure connection to websites shows up in red in some Excel spreadsheet. We can't have Excel spreadsheets showing red in fintech. /s
account42 11mo ago
> Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously? I’d wager that’s almost at the whatever is the statistics equivalent of the Planck length level of probability. It doesn't matter if every certificate authority is compromised or just one. One is all that is needed to sign certificates for all websites.
- mark_round 11mo ago
  Author here, hi! Was just venting last night, but that's a very good point, I'll update it later with your correction :)
  - acer4666 11mo ago
    You should make it about CT logs. I believe you need to compromise at least three of them.
    - tialaramex 11mo ago
      The whole point of the logs is that they're tamper-evident. If you think the certificate you've seen wasn't logged you can show proof. If you think the logs tell you something different from everybody else you can prove that too. It is striking that we don't see that. We reliably see people saying "obviously" the Mossad or the NSA are snooping but they haven't shown any evidence that there's tampering
      - rnhmjoj 11mo ago
        > It is striking that we don't see that It probably just means they are asking the providers to hand over the data, no need to perform active attacks.
      - dns_snek 11mo ago
        > We reliably see people saying "obviously" the Mossad or the NSA are snooping but they haven't shown any evidence that there's tampering Why would they use the one approach that leaves a verifiable trace? That'd be foolish. - They can intercept everything in the comfort of Cloudflare's datacenters - They can "politely" ask Cloudflare, AWS, Google cloud, etc. to send them a copy of the private keys for certificates that have already been issued - They either have a backdoor, or have the capability to add a backdoor in the hardware that generates those keys in the first place, should more convenient forms of access fail.
        11mo ago
sroussey 11mo ago
Lame on user machines, but sometimes needed in a server environment. Easier to detect if someone is hauling off with your database as that will be the one you can’t see what’s going on. Of course, solve one problem and introduce three more.
pimterry 11mo ago
It's definitely annoying if you work in enterprise, but on the flip side: the fact that these enterprise requirements exist is the main reason that TLS certificate configurability is possible at all, without which it would be dramatically harder (or impossible) to reverse engineer or do security & privacy research on mobile apps, IoT, etc etc etc. Enterprise control over company devices and user control over personal devices are not so different. A few apps do use certificate pinning nowadays, which creates similar problems, but saying "you can never add your own MitM TLS cert" is not far from certificate pinning everything everywhere all the time. Good luck creating a new home assistant integration for your smart airfryer when you can't read any of the traffic from its app. Imo: let's make it easier! Standardize TLS configuration for all tools, make easy cert configuration of devices a legal requirement (any smart device sold with hardcoded CA certificates is a device with a fixed end date, where the CA certs expire and it becomes a brick), guarantee user control over their own TLS trust, and provide good tools to check exactly who you're trusting (and expose that clearly to users). Not really practical of course (and opens all sorts of risky games with nation state interception as well) but there are upsides here as well.
- acdha 11mo ago
  > Standardize TLS configuration for all tools, make easy cert configuration of devices a legal requirement I think this is the right idea (it’s configuring dozens of things which causes problems) but the other idea I’d consider is standardizing a key escrow mechanism where the session keys could be exported to a monitoring server. That avoids needing active interception with all of the problems that causes, and would pair well with a standardized OS-level warning that all communications are monitored by «name from the monitor cert» which the corporate types are required to display anyway.
samuel 11mo ago
I agree with the sentiment, but I think it's a pretty naive view of the issue. Companies will want all info they can in case some of their workers does something illegal-inappropiate to deflect the blame. That's a much more palpable risk than "local CA certificates being compromised or something like that. And some of the arguments are just very easily dismissed. You don't want your employer to see you medical records? Why were you browsing them during work hours and using your employers' device in the first place?
- immibis 11mo ago
  In Europe they prefer not to go to jail for privacy violations. It turns out most of these "communist" regulations are actually pretty great.
  - johncolanduoni 11mo ago
    Does GDPR (or similar) establish privacy rights to an employee’s use of a company-owned machine against snooping by their employer? Honest question, I hadn’t heard of that angle. Can employers not install EDR on company-owned machines for EU employees?
    - apexalpha 11mo ago
      Yes, at least in the Netherlands it is generally accepted that employees can use your device personally, too. Using a device owned by your company to access your personal GMail account does NOT void your legal right to privacy.
      - johncolanduoni 11mo ago
        So does nobody in Europe use an EDR or intercepting proxy since GDPR went into force?
        samuel 11mo ago
        I have found a definite answer from the Dutch Protection Agency (although it could be out of date). https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf https://english.ncsc.nl/binaries/ncsc-en/documenten/factshee...
gschizas 11mo ago
The fact that most tools have completely different ways to allow them to add certificates is the biggest pain. Git, Python and Rust also have large issues. Git doesn't default to "http.schannel". Python (or rather requests, or maybe urllib3) only looks at its own certificate store, and I have no idea how Rust does this (well, I use uv, and it has its own problems - I know about the --use-native-tls flag, but it should be a default at the least).
- nikanj 11mo ago
  And many things break in different, exciting ways. For example, we discovered that whilst the JVM can be configured to use system certificate store, that does not apply to websocket connections. So the product seems to be able to connect to the server, but all socket connections bork out with TLS errors. Fun! And so many of those products deliver broken chains, and your client needs to download more certificates transparently ( https://systemweakness.com/the-hidden-jvm-flag-that-instantly-fixes-ssl-errors-10f27c1dffe7 https://systemweakness.com/the-hidden-jvm-flag-that-instantl... ) Double the fun!
- sureglymop 11mo ago
  I have this similar gripe when it comes to http proxy configuration. It's invisible to you until you are in an execution environment where you are mandated to use the providers proxy configuration. Some software reads "expected" env variables for it, some has its own config or cli flags, most just doesn't even bother/care about supporting it.
  - amiga386 11mo ago
    Chiefly because "supporting it" requires a full JavaScript interpreter, and subscribing to changes in "system settings" during the lifetime of your program. Easier just to support http_proxy/https_proxy/no_proxy (and what standard for no_proxy? Does it support CIDR ranges?) or even less flexibility than that.
    - sureglymop 11mo ago
      If only http_proxy/https_proxy/no_proxy at startup time were more widely supported then. In my case I had to deploy software into a kubernetes cluster managed by a different company that required these configurations.
Daviey 11mo ago
Honestly, the author is spot on about the normalisation problem. I've watched this play out at multiple organisations. You implement TLS inspection, spend ages getting certs deployed, and within six months `curl -k` is in half your runbooks because "it's just the corporate proxy again". He's also absolutely right about the architectural problems too, single points of failure, performance bottlenecks, and the complexity in cloud-native environments. That said, it can be a genuinely valuable layer in your security arsenal when done properly. I've seen it catch real threats, such as malware C2 comms, credential phishing, data exfiltration attempts. These aren't theoretical; they happen daily. Combined with decent threat intelligence feeds and behavioural analytics, it does provide visibility that's hard to replicate elsewhere. But, and this is a massive but, you can't half-arse it. If you're going to do TLS inspection, you need to actually commit: Treat that internal CA like it's the crown jewels. HSMs, strict access controls, proper rotation schedules, full-chain and sensible life-span. The point about concentrated risk is bang on, you've turned thousands of distributed CA keys into one single target. So act like it. Run it like a proper CA with proper key signing ceremonies and all the safeguards etc. Actually invest in proper cert distribution. Configuration management (Ansible/Salt/whatever), golden container base images with the CA bundle baked in, MDM for endpoints, cloud-init for VMs. If you can't reliably push a cert bundle to your entire estate, you've got bigger problems than TLS inspection. Train people properly on what errors are expected vs "drop everything and call security". Document the exceptions. Make reporting easy. Actually investigate when someone raises a TLS error they don't recognise. For dev's, it needs to just work without them even thinking about it. Then they don't need to work around it, ever. If they need to, the system is busted. Scope it ruthlessly. Not everything needs to go through the proxy. Developer workstations with proper EDR? Maybe exclude them. Production services with cert pinning? Route direct. Every blanket "intercept everything" policy I've seen has been a disaster. Particularly for end-users doing personal banking, medical stuff, therapy sessions, do you really want IT/Sec seeing that? Use it alongside modern defences. ie EDR, Zero Trust, behavioural analytics, CASB. It should be one layer in defence-in-depth, not your entire security strategy. Build observability, you need metrics on what's being inspected, what's bypassing, failure rates, performance impact. If you can't measure it, you can't manage it. But Yeah, the core criticism stands though, even done well, it's a massive operational burden and it actively undermines trust in TLS. The failure modes are particularly insidious because you're training people to ignore the very warnings that are meant to protect them. The real question isn't "TLS inspection: yes or no?" It's: "Do we have the organisational maturity, resources, and commitment to do this properly?" If you're not in a regulated industry or don't have dedicated security teams and mature infrastructure practices, just don't bother. But if you must do it, and plenty of organisations genuinely must, then do it properly or don't do it at all.
nikanj 11mo ago
Companies know that it's important to have Cybersecurity™. A vendor shows up with shiny brochures, and company is happy to purchase Cybersecurity™. Now they don't have to worry about it anymore, they bought a product that sits in the corner and delivers Cybersecurity™
- Nextgrid 11mo ago
  You've perfectly summarized the entire industry. There's no actual market pressure to be secure, so nobody cares about threat modeling, cost/benefit of security solutions, etc. The only pressure in case of breach is political blame that you need to deflect. The point of a cybersecurity solution is to be there, remind you it is there, and allow you to deflect blame in case of disaster. Whether it actually increases security is merely a bonus side-effect.
KaiserPro 11mo ago
Our cyber team have installed zscaler on most people's laptop, and somewhere in the fabric of the office internet connection.[1] For those that don't know, its a MITM proxy with certificates so that it can inspect and unroll TLS traffic. ostensibly its there to stop data exfiltration, as we've had a number of incidents where people have stolen data and sent it to competitors. (our c-suite don't have as much cyber shit installed, despite them being the ones that are both targets more, and broken the rules more....) Now, I don't like zscaler, and I can sorta see the point of it. But. Our cyber team is not a centre of technical excellence. They somehow managed to configure zscaler to send out the certs for a random property company, when people were trying to sign into our VPN. this broke loads of shit and made my team (infra) look bad. The worrying part is they still haven't accepted that serving a random property company's website cert instead of our own/AWS's cert is monster fuckup, and that we need to understand _why_ that happened before trying anything again. [1] this makes automatic pen testing interesting because everything we scan has vulnerabilities for NFS/CIFS, FTP and TCP dns.
- a012 11mo ago
  Security team in most of the corporates is just a bunch of checklists markers, so for zscaler, crowdstrike or whatever they’re doing for compliance and/or certification and you can’t say no to it because it’s the company policy and who know better than “security” team?
  - devdp430 11mo ago
    This is 100% true.
  - lisbbb 11mo ago
    The problem I have with them is they are a no-talent crowd who enjoy pushing everyone around.
NicolaiS 11mo ago
Got acquired by a Fortune 500 and recieved new laptop. First hour I'm seeing TLS errors everywhere except the browser. They'd half-baked their internal CA rollout, so wasn't trusted properly. By day two I started validating their setup. The CA literally had a typo in the company name, not a great sign. A quick check with badssl.com showed that any self-signed(!) cert was being transparently MITM'ed and re-signed by their trusted corporate cert. Took them 40 days to fix it. Another fun side-effect of this is that devs will just turned off TLS verification, so their codebase is full of `curl -k`, `verify_mode = VERIFY_NONE`, `ServerCertificateValidationCallback = () => true`, ... Exactly the thing you want to see at a big fintech company /s
- lisbbb 11mo ago
  I've experienced similar. It has definitely made me less enthusiastic about working for any of those fools ever again. It's all just an exercise in mediocrity. The illiterate emails people send out are even worse--I swear that a lot of US born adults are functionally illiterate.
arianvanp 11mo ago
Complains about TLS inspection, yet fronts their website on the biggest and most widely deployed TLS introspection middle box in the world ... Why do we all disdain local TLS inspection software yet half the Internet terminates their TLS connection at Cloudflare who are most likely giving direct access to US Intelligence? It's so much worse as it's infringing on the privacy and security of billions of innocent people whilst inspection software only hurts some annoying enterprise folks. I wish we all hopped off the Cloudflare bandwagon.
- phito 11mo ago
  I wish so too, same for all the self-hosters using tailscale...
  - kreetx 11mo ago
    These are not the same thing, the parent is confused..
  - dns_snek 11mo ago
    Tailscale connections don't get terminated by a middle box, it's just end-to-end encrypted Wireguard under the hood. Cloud-hosted control panel is a risk because they could push malicious configuration changes to your clients (ACLs and new nodes if you're not using the lock feature), but they can't do it without leaving a trace like Cloudflare can.
  - progbits 11mo ago
    Tailscale cannot passively observe traffic. They could inject malicious keys into your config but would be hard to mask the evidence of that.
    - treesknees 11mo ago
      Would it be hard? I thought the point of tailscale was not having to manage or concern yourself with key distribution.
apexalpha 11mo ago
I largely agree with the author. When our SOC wanted to implement TLS inspection I blocked it. Mostly because we not nearly at the security level for this, but also because it just fucks with so many things. That said, we are not a business dealing with highly sensitive data or legal responsibilities surrounding data loss prevention. If you are a business like that, say a bank or a hospital, you want to be able to block patient / customer data leaving your systems. You can do this by setting up a regex for a known format like patient numbers or bank account numbers. This requires TLS inspection obviously. Though this makes it harder to steal this data, not impossible. It does however allow the C-suite to say they did everything they could to prevent it.
- apexalpha 11mo ago
  Oh and the software (Netskope) was only able to decrypt our traffic in the cloud. Lmao not in a million fucking years will I upload our data to an American company in fucking plaintext.
  - dcminter 11mo ago
    Netskope and the other DLP tools at my last gig would completely lock up my network connection for around 30 seconds every hour or two while maxing out 100% of a core. Fun times. The issue was still there a year after I first encountered it so I have grave doubts about the competence of those vendors. On the other hand I am sympathetic to the needs of big regulated orgs to show they're doing something to avoid data loss. It's a painful situation.
franga2000 11mo ago
I agree with the sentiment, but this part is complete bullshit: > what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously Who cares? It's not like all CAs would have to be breached, just one. CA certs are not scoped, so the moment one CA gets breached, we're all fucked. CT helps, but AFAIK it's still not enforced everywhere yet
nubinetwork 11mo ago
This is why I don't use the computers at work for anything not work related. They've been spying on us for at least 10 years.
nly 11mo ago
"If you use my (private) network you follow my rules" And I find it hard to argue with that. I've been using a VPN habitually on my phone and my (personal) laptop for a decade now. Work, home, travel. Doesn't matter. It's always on.
- redrix 11mo ago
  How do you find your typical daily battery life with it always on? I’ve tried this in the past and had to revert as I found it made a noticeable difference in my day-to-day. Curious to hear the experience of others.
  - hexbin010 11mo ago
    Was it OpenVPN that you tried in the past? Wireguard seems much better
  - letmetweakit 11mo ago
    I have the impression tailscale drains my battery on macOS and iOS, only turn it on when truly needed.
    - newdee 11mo ago
      Yeah, it most certainly does. Very noticeable on iOS. I don’t know if this is an Apple specific thing, or if it’s a similar story on Android. It’s WireGuard underneath, which is designed to not be very chatty when idle, so I’d put this down to regular back and forth with Tailscale’s control plane, relays, etc. It’s a shame really, because a huge value prop of TS is that it’s a VPN you just leave on and forget about. I hate having to toggle it when I inevitably forget to and wonder why I’m getting connection errors to private resources.
  - ectospheno 11mo ago
    I have an ipv6 wireguard vpn from my iOS phone to my home network. It routes all traffic through my home isp. I use the wireguard iOS client. Battery life has been fine for me. One caveat is that background updates are disabled for almost every app.
Wowfunhappy 11mo ago
I work for a school. My traffic is not MITM'd, but the kids' traffic is, because we don't want them using their school-issued laptops to play games or go shopping, and you can't adequately block stuff if it's all encrypted.
- TheChaplain 11mo ago
  I can't imagine the headache as a school when parents come yelling "why did you allow my child on site XXX?!"
- lousken 11mo ago
  Whitelists instead of blacklists?
  - Wowfunhappy 11mo ago
    I'm pretty sure we'd still need to break TLS. Domain-level just isn't granular enough.
    - lousken 11mo ago
      I still find that dumb that you even need to do that. Machines especially for schools should be able to have software policies set directly on them to limit such sites. I don't know how much chromeOS is configurable and if you can e.g. force it to only use specific network and network interface, or if a student can connect it to a different network somehow, because it would be kinda pointless otherwise.
      - Wowfunhappy 11mo ago
        The school-issued laptops are all Macbooks. To be clear I'm not in the IT department so I don't know exactly what the setup is, but I see my students using their computers. A VPN is involved, which is what made me assume they are doing TLS shenanigans—I guess I could theoretically be wrong, but it's definitely more granular than domain-level blocking, so I don't know how else it could work. The computers connect to this VPN automatically on startup. In the moments before the VPN connects, the internet does not work. > Machines especially for schools should be able to have software policies set directly on them to limit such sites. It's a good point—if you just did this client-side instead of on the network level, you wouldn't have to deal with TLS or anything. It seems clear to me that they aren't doing that (given the VPN) and it's not immediately obvious to me why.
kotaKat 11mo ago
zScaler is a load of shit, especially with some of its absolutely dumb policies like “malicious TLDs”. Because the Framework laptop site at frame.work is malicious, of course. God, I love CURLing crap from my workstation and not getting the files I needed but instead a bunch of mangled HTML telling me zScaler was going to scan what I was going to download. Bonus points that it puts me in the wrong country because I’m closer to Montreal than any American locations so half the time I’m stuck in French Canadian on the web from my New York office. Triple bonus points that I’m required to test speed at client sites and zScaler completely mangles our presentable results. Quadruple bonus points that I put in "because I feel like it" into every elevation request I make on my corporate machine and our "cyber team" has literally never looked at elevation reports to ask what the hell I'm doing...
cornonthecobra 11mo ago
> Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously? Considering that CloudFlare has managed to MitM a huge part of the internet, I'd say that probability is not just non-zero, but greater than by a worrying margin.
- acdha 11mo ago
  That’s not what MITM means, and also misunderstands how CAs work. Cloudflare is a concern for how many people would be affected if there was another Cloudbleed but misstating their relationship with their customers isn’t going to accomplish anything.
  - JetSpiegel 11mo ago
    How is that not a MITM? Just because it's the modern day CryptoAG?
    - acdha 11mo ago
      Because it’s not an attack but rather a voluntary infrastructure choice by a company. We don’t say that Varnish is a MITM because it’s in front of my application, because it’s intentional and under my control. Misusing the term muddies the topic rather than adding clarity, and while there’s a very useful discussion about centralization or why Cloudflare’s most stringent customers might want to deploy their Keyless SSL service that discussion won’t happen if someone misuses the term.
      - JetSpiegel 11mo ago
        Yes, but Cloudflare is not under your control, that's the point.
kreetx 11mo ago
The title should say "Stop inspecting TLS", the current title reads like the TLS standard or technology is modified in a way to not work properly.
- Wowfunhappy 11mo ago
  For what it's worth, I knew exactly what this was going to be about before I clicked.
voidUpdate 11mo ago
I'm hoping this doesn't apply to things like Fiddler, because without the ability to see what's actually coming over the wire with a https connection, things can be a nightmare to debug sometimes
bandie91 11mo ago
personally i'm happy that i can MITM my docker when it wants to pull gigs of images the 1000th time upstream and just serve them from a local OCI cache server instead.
- mirashii 11mo ago
  You don't need to MITM docker for this, you can just configure your containerd or equivalent backend properly.
  - baobun 11mo ago
    How about that build script cloning the same git repo or wgetting the same release bundle over and over?
dminuoso 11mo ago
One thing that has not quite been mentioned in the blog, is how much of the MITM spyware comes from very big and well known „security“ companies. You know, the ones that really know about security. X-PAN-AUTHCHECK type of security. The amount of CVEs some of the big firewall companies collect make it seem like it is a competition for the poorest security hygiene. The real problem we have is compliance theatre where someone in management forces these solutions onto their IT department just so they can check a box on their sheets and shift all responsibilities away.
YesThatTom2 11mo ago
More and more big customers (especially banks) are requiring this kind of self-inflicted-MITM attack from all their suppliers. Do you want to have customers? Get ready for zscaler! How do you propose compliance with their exfiltration protection requirements? (And “turn down $ from those customers” is not an answer)
- deleted 11mo ago
  [deleted]
lousken 11mo ago
As a sysadmin I also hate this. Instead, I do block stuff based on DNS requests and I also block any other DNS provider as well as malicious IPs. At this point in time, Microsoft is the bigger enemy here - some of their policies are just insane and none of this MITM will help [0][1] [0] https://www.microsoft.com/en-us/microsoft-365/roadmap?id=490064 https://www.microsoft.com/en-us/microsoft-365/roadmap?id=490... [1] https://techcommunity.microsoft.com/blog/microsoft365copilotblog/employees-can-bring-copilot-from-their-personal-microsoft-365-plans-to-work---wh/4458212 https://techcommunity.microsoft.com/blog/microsoft365copilot...
sqbic 11mo ago
What changed my mind to be in favor of TLS inspection at work environments was seeing what kind of highly confidential stuff employees might be copy-pasting to random websites, LLM assistants, cloud-based "desktop applications" and such against the approved use policies of each of these tools without giving it a second thought. TLS inspection products can intercept the paste transaction before the data leaves the company network, hitting the user with a "No you didn't! Shame on you!"-banner and notify the admins how a user just tried to paste hundreds of customers' personal information and credit card details into some snooping website, or into otherwise allowed LLM chat which still is not allowed to be used with confidential information. There can even be automations to lock the user/device out immediately if something like this is going on, be it the user or some undetected malware in the user's device attempting the intercepted action. Being able to do these kinds of very specifically targeted interceptions can prevent potentially huge disasters from happening while still allowing users more freedom in taking advantage of the huge variety of productivity tools available these days. No need to choose between completely blocking all previously unseen tools or living in fear of disastrous leaks when there are fine-grained possibilities to control what kind of information can be fed to the tools and from where. There are plenty of organizations out there where it is completely justified to enforce such limitations and monitoring in company devices. Policies can forbid personal use entirely where it is deemed necessary and legal to do so. Of course the policies and the associated enforced monitoring needs to be clearly communicated and there needs to be carefully curated configurations to control where and how TLS is or isn't intercepted so employee privacy laws and regulations aren't breached either.
- iso1631 11mo ago
  So deploy end point security, which sits in the kernel and can thus access the unencrypted communication
  - treesknees 11mo ago
    Aren’t most TLS implementations still using things like OpenSSL in userspace? How would the kernel get access to the request?
iamcalledrob 11mo ago
This kind of TLS "man in the middle" tech is so frustrating to deal with, because it ends up breaking things. For example, I've encountered zscaler setups in the wild which close TLS connections if non-HTTP traffic is encountered. Presumably the traffic inspection fails since there is no HTTP request, and this failure path closes the socket. It's hard to say whether it's due to the customer's IT dept's config, or zscaler itself -- but as far as the customer is concerned, it's my problem.
wolfi1 11mo ago
a company, I worked for, had their own endpoint which you can easily introduce in windows, unfortunately every other tls connection which does not use the windows certificate store breaks because of that, so maven, npm et al won't work
zmmmmm 11mo ago
I'd like to know if there is any real world data about the effectiveness of TLS inspection in preventing harm. It's empirically a massive tax on any organisation that engages in any kind of software engineering or technical work. I estimate that something like 3% of all our engineering effort goes to working around the TLS inspection which breaks security on literally every piece of infrastructure we build. And it embeds all the harm that the article alludes to (and more). So it takes quite a significant balance of upside to counteract that. So is the benefit worth it? Is there data to prove it? Or is it just authoritarian IT departments drunk on power implementing this stuff? I'd love to know.
1970-01-01 11mo ago
Isn't the accepted risk of losing all the digital keys equal to the risk of someone booting up the root CA and walking away with it? I can see that happening in a junta situation. https://www.iana.org/dnssec/ceremonies https://www.iana.org/dnssec/ceremonies
acdha 11mo ago
I really wish more places would enable explicit proxies: if you have a mandate to inspect all traffic, block 443 except to proxy.megacorp.com and configure clients to use it. You lose all of the bugs and security issues caused by the security software—fun fact, Palo Alto _still_ doesn’t correctly implement TLS 1.2!—which as the author points out is basically training users to disable validation or ignore errors. This is so common I’ve seen security people with the usual certs giving advice from ChatGPT to add -k to curl calls, which is especially great when that’s baked into a container which will run on other networks. One really nice win is troubleshooting: inspection appliances break decades of performance and reliability work which groups like the Linux kernel developers have done, and they make all errors have the same symptom so the security team will now be the only people who can troubleshoot network-level issues. Explicit proxying can’t fix your network but it makes it very clear where the problem needs to be investigated.
- rini17 11mo ago
  Are you aware how https proxying works? Clients use CONNECT method and after that everything is opaque to the proxy. So without mitm you only know remote IP address.
  - acdha 11mo ago
    There’s another difference which is what I was referring to: in both cases, your proxy has to forge the SSL certificate for the remote server but in the transparent case it also must intercept network traffic intended for the remote IP. That means that clients can’t tell whether an error or performance problem is caused by the interception layer or the remote server (sometimes Docker Hub really is down…) and it can take more work for the proxy administrator to locate logs. If you explicitly configure a proxy, the CONNECT method can trigger the same SSL forgery but because it’s explicit the client’s view is more obvious and explainable. If my browser gets an error connecting to proxy.megacorp.com I don’t spend time confirming that the remote service is working. If the outbound request fails, I’ll get a 5xx error clearly indicating that rather than having to guess at what node dropped a connection or why. This also provides another way to implement client authentication which could be useful if you have user-based access control policies. It’s not a revelation, but I think this is one of the areas where trying to do things the easy way ends up being harder once you factor in frictional support costs. Transparent proxying is trading a faster rollout for years of troubleshooting.
lisbbb 11mo ago
I'm glad someone is speaking out about this. When I first found out that the major Fortune 500 company I was working for did this, I was happy I never did any online banking or any personal finance logins using the work computer. It's obscene, quite frankly. I have many criticisms of big-corpo security and how overbearing and paranoid it has become such that developers can barely get a thing done in a lot of these big firms. Plus, most of the internal threats are embezzlers who get away with it (for awhile, at least). I did work for one place that had a Chinese national attempt to make off with the entire customer database, but he did it by burning CDs (circa 2006).
dajt 11mo ago
This is a massive pain at my work, same as I'm sure most other comments are saying. I use Platform.IO for firmware development and can't build my firmware unless I hotspot my phone. I would say that's a PIO bug unless there is a flag I don't know about, but it's exposed by this nuisance of a firewall. The devs where I am spend much of our time hotspotted to our phones with the corporate network never connected so IP goes out over the mobile network. Whenever possible we use 'do not verify server certs' flags in libs and commands which is not ideal.
parliament32 11mo ago
The author is complaining a lot about implementation pains without taking a step back and looking at why it exists in the first place. Say you work at a place that deals with credit cards. You, as a security engineer, have a mandate to stop employees from shipping CC numbers outside the org. You can educate all you want, you can have scary policies and HR buy-in, you can have all the "Anomaly detection, Zero Trust network architecture, EDR, Netflow analysis" in the world, but exactly zero of those will stop Joe Lunchbox from copy/pasting a block with a CC number in the middle into ChatGPT. You know what will? A TLS-inspecting proxy with some DLP bits and bobs. It sucks, yes. But it works, and (short of the fool's errand of trying to whitelist every site any employee needs) it's the only thing that works. And yes, I'm aware PCI DSS has additional requirements for CDEs and whatnot, but really this can apply to anything -- a local government office dealing with SSNs, a school with student ID numbers, a corporation with trade secrets.. these problems exist everywhere, and implementing PCI-like controls is often a bridge too far for unregulated industries.
- dadrian 11mo ago
  That is not true, you can run DLP on an endpoint directly and inside a browser directly (e.g. via an extension or direct integration hooks). You can also try to stop the situation where the CC numbers are in the clear anywhere in the first place, so that you can't copy/paste them around. What happens if someone writes the CC number down on a piece of paper?
  - parliament32 11mo ago
    Endpoint DLP helps but it's not even close to bulletproof. Just for fun, if you have DLP at work, open the integrated browser in VS Code and notice how you can send protected test strings without anything chirping you. > CC numbers are in the clear anywhere in the first place Sounds great in theory, until you realize that in a large number of industries the majority of employees need access to protected data to do their jobs. Imagine telling the IRS their employees can't see/use cleartext SSNs. As for paper / mobile phones / whatever.. you're not wrong, but physical security is typically someone else's job.

175 points by todsacerdoti 11mo ago

But I need to see what they are googling! /sarcasm

I remember at my first job, the internet stopped working at my workstation. I got on the phone with IT, and the guy said "looks like you don't have our new certificates." I asked why I would need my employer's certificates. He said "because we MITM every connection." I asked if that was even legal, and he said yes it's legal. At another job I was handling a support ticket where a customer was asking, in so many words, "can I get HTTP headers of requests flowing through my Envoy TLS reverse proxy?" I said that they could terminate TLS at the proxy and redo things that way, but then that wouldn't be a TLS proxy it'd be a MITM or a gateway. They could log the downstream/upstream and duration of connections, but that wouldn't help.

cxr 11mo ago
No one who understands what "MITM" means should have any expectation that I/O with a device owned and administered by a third party can be trusted (whether they do it by subverting PKI with internal certificate or not).

phoronixrly 11mo ago

Hey, allowing your employees to have secure connection to websites shows up in red in some Excel spreadsheet. We can't have Excel spreadsheets showing red in fintech. /s

account42 11mo ago

> Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously? I’d wager that’s almost at the whatever is the statistics equivalent of the Planck length level of probability. It doesn't matter if every certificate authority is compromised or just one. One is all that is needed to sign certificates for all websites.

mark_round 11mo ago
Author here, hi! Was just venting last night, but that's a very good point, I'll update it later with your correction :)
- acer4666 11mo ago
  You should make it about CT logs. I believe you need to compromise at least three of them.
  - tialaramex 11mo ago
    The whole point of the logs is that they're tamper-evident. If you think the certificate you've seen wasn't logged you can show proof. If you think the logs tell you something different from everybody else you can prove that too. It is striking that we don't see that. We reliably see people saying "obviously" the Mossad or the NSA are snooping but they haven't shown any evidence that there's tampering
    - rnhmjoj 11mo ago
      > It is striking that we don't see that It probably just means they are asking the providers to hand over the data, no need to perform active attacks.
    - dns_snek 11mo ago
      > We reliably see people saying "obviously" the Mossad or the NSA are snooping but they haven't shown any evidence that there's tampering Why would they use the one approach that leaves a verifiable trace? That'd be foolish. - They can intercept everything in the comfort of Cloudflare's datacenters - They can "politely" ask Cloudflare, AWS, Google cloud, etc. to send them a copy of the private keys for certificates that have already been issued - They either have a backdoor, or have the capability to add a backdoor in the hardware that generates those keys in the first place, should more convenient forms of access fail.
      - 11mo ago

sroussey 11mo ago

Lame on user machines, but sometimes needed in a server environment. Easier to detect if someone is hauling off with your database as that will be the one you can’t see what’s going on. Of course, solve one problem and introduce three more.

pimterry 11mo ago

It's definitely annoying if you work in enterprise, but on the flip side: the fact that these enterprise requirements exist is the main reason that TLS certificate configurability is possible at all, without which it would be dramatically harder (or impossible) to reverse engineer or do security & privacy research on mobile apps, IoT, etc etc etc. Enterprise control over company devices and user control over personal devices are not so different. A few apps do use certificate pinning nowadays, which creates similar problems, but saying "you can never add your own MitM TLS cert" is not far from certificate pinning everything everywhere all the time. Good luck creating a new home assistant integration for your smart airfryer when you can't read any of the traffic from its app. Imo: let's make it easier! Standardize TLS configuration for all tools, make easy cert configuration of devices a legal requirement (any smart device sold with hardcoded CA certificates is a device with a fixed end date, where the CA certs expire and it becomes a brick), guarantee user control over their own TLS trust, and provide good tools to check exactly who you're trusting (and expose that clearly to users). Not really practical of course (and opens all sorts of risky games with nation state interception as well) but there are upsides here as well.

acdha 11mo ago
> Standardize TLS configuration for all tools, make easy cert configuration of devices a legal requirement I think this is the right idea (it’s configuring dozens of things which causes problems) but the other idea I’d consider is standardizing a key escrow mechanism where the session keys could be exported to a monitoring server. That avoids needing active interception with all of the problems that causes, and would pair well with a standardized OS-level warning that all communications are monitored by «name from the monitor cert» which the corporate types are required to display anyway.

samuel 11mo ago

I agree with the sentiment, but I think it's a pretty naive view of the issue. Companies will want all info they can in case some of their workers does something illegal-inappropiate to deflect the blame. That's a much more palpable risk than "local CA certificates being compromised or something like that. And some of the arguments are just very easily dismissed. You don't want your employer to see you medical records? Why were you browsing them during work hours and using your employers' device in the first place?

immibis 11mo ago
In Europe they prefer not to go to jail for privacy violations. It turns out most of these "communist" regulations are actually pretty great.
- johncolanduoni 11mo ago
  Does GDPR (or similar) establish privacy rights to an employee’s use of a company-owned machine against snooping by their employer? Honest question, I hadn’t heard of that angle. Can employers not install EDR on company-owned machines for EU employees?
  - apexalpha 11mo ago
    Yes, at least in the Netherlands it is generally accepted that employees can use your device personally, too. Using a device owned by your company to access your personal GMail account does NOT void your legal right to privacy.
    - johncolanduoni 11mo ago
      So does nobody in Europe use an EDR or intercepting proxy since GDPR went into force?
      - samuel 11mo ago
        I have found a definite answer from the Dutch Protection Agency (although it could be out of date). https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf https://english.ncsc.nl/binaries/ncsc-en/documenten/factshee...

gschizas 11mo ago

The fact that most tools have completely different ways to allow them to add certificates is the biggest pain. Git, Python and Rust also have large issues. Git doesn't default to "http.schannel". Python (or rather requests, or maybe urllib3) only looks at its own certificate store, and I have no idea how Rust does this (well, I use uv, and it has its own problems - I know about the --use-native-tls flag, but it should be a default at the least).

nikanj 11mo ago
And many things break in different, exciting ways. For example, we discovered that whilst the JVM can be configured to use system certificate store, that does not apply to websocket connections. So the product seems to be able to connect to the server, but all socket connections bork out with TLS errors. Fun! And so many of those products deliver broken chains, and your client needs to download more certificates transparently ( https://systemweakness.com/the-hidden-jvm-flag-that-instantly-fixes-ssl-errors-10f27c1dffe7 https://systemweakness.com/the-hidden-jvm-flag-that-instantl... ) Double the fun!
sureglymop 11mo ago
I have this similar gripe when it comes to http proxy configuration. It's invisible to you until you are in an execution environment where you are mandated to use the providers proxy configuration. Some software reads "expected" env variables for it, some has its own config or cli flags, most just doesn't even bother/care about supporting it.
- amiga386 11mo ago
  Chiefly because "supporting it" requires a full JavaScript interpreter, and subscribing to changes in "system settings" during the lifetime of your program. Easier just to support http_proxy/https_proxy/no_proxy (and what standard for no_proxy? Does it support CIDR ranges?) or even less flexibility than that.
  - sureglymop 11mo ago
    If only http_proxy/https_proxy/no_proxy at startup time were more widely supported then. In my case I had to deploy software into a kubernetes cluster managed by a different company that required these configurations.

Daviey 11mo ago

Honestly, the author is spot on about the normalisation problem. I've watched this play out at multiple organisations. You implement TLS inspection, spend ages getting certs deployed, and within six months `curl -k` is in half your runbooks because "it's just the corporate proxy again". He's also absolutely right about the architectural problems too, single points of failure, performance bottlenecks, and the complexity in cloud-native environments. That said, it can be a genuinely valuable layer in your security arsenal when done properly. I've seen it catch real threats, such as malware C2 comms, credential phishing, data exfiltration attempts. These aren't theoretical; they happen daily. Combined with decent threat intelligence feeds and behavioural analytics, it does provide visibility that's hard to replicate elsewhere. But, and this is a massive but, you can't half-arse it. If you're going to do TLS inspection, you need to actually commit: Treat that internal CA like it's the crown jewels. HSMs, strict access controls, proper rotation schedules, full-chain and sensible life-span. The point about concentrated risk is bang on, you've turned thousands of distributed CA keys into one single target. So act like it. Run it like a proper CA with proper key signing ceremonies and all the safeguards etc. Actually invest in proper cert distribution. Configuration management (Ansible/Salt/whatever), golden container base images with the CA bundle baked in, MDM for endpoints, cloud-init for VMs. If you can't reliably push a cert bundle to your entire estate, you've got bigger problems than TLS inspection. Train people properly on what errors are expected vs "drop everything and call security". Document the exceptions. Make reporting easy. Actually investigate when someone raises a TLS error they don't recognise. For dev's, it needs to just work without them even thinking about it. Then they don't need to work around it, ever. If they need to, the system is busted. Scope it ruthlessly. Not everything needs to go through the proxy. Developer workstations with proper EDR? Maybe exclude them. Production services with cert pinning? Route direct. Every blanket "intercept everything" policy I've seen has been a disaster. Particularly for end-users doing personal banking, medical stuff, therapy sessions, do you really want IT/Sec seeing that? Use it alongside modern defences. ie EDR, Zero Trust, behavioural analytics, CASB. It should be one layer in defence-in-depth, not your entire security strategy. Build observability, you need metrics on what's being inspected, what's bypassing, failure rates, performance impact. If you can't measure it, you can't manage it. But Yeah, the core criticism stands though, even done well, it's a massive operational burden and it actively undermines trust in TLS. The failure modes are particularly insidious because you're training people to ignore the very warnings that are meant to protect them. The real question isn't "TLS inspection: yes or no?" It's: "Do we have the organisational maturity, resources, and commitment to do this properly?" If you're not in a regulated industry or don't have dedicated security teams and mature infrastructure practices, just don't bother. But if you must do it, and plenty of organisations genuinely must, then do it properly or don't do it at all.

nikanj 11mo ago

Companies know that it's important to have Cybersecurity™. A vendor shows up with shiny brochures, and company is happy to purchase Cybersecurity™. Now they don't have to worry about it anymore, they bought a product that sits in the corner and delivers Cybersecurity™

Nextgrid 11mo ago
You've perfectly summarized the entire industry. There's no actual market pressure to be secure, so nobody cares about threat modeling, cost/benefit of security solutions, etc. The only pressure in case of breach is political blame that you need to deflect. The point of a cybersecurity solution is to be there, remind you it is there, and allow you to deflect blame in case of disaster. Whether it actually increases security is merely a bonus side-effect.

KaiserPro 11mo ago

Our cyber team have installed zscaler on most people's laptop, and somewhere in the fabric of the office internet connection.[1] For those that don't know, its a MITM proxy with certificates so that it can inspect and unroll TLS traffic. ostensibly its there to stop data exfiltration, as we've had a number of incidents where people have stolen data and sent it to competitors. (our c-suite don't have as much cyber shit installed, despite them being the ones that are both targets more, and broken the rules more....) Now, I don't like zscaler, and I can sorta see the point of it. But. Our cyber team is not a centre of technical excellence. They somehow managed to configure zscaler to send out the certs for a random property company, when people were trying to sign into our VPN. this broke loads of shit and made my team (infra) look bad. The worrying part is they still haven't accepted that serving a random property company's website cert instead of our own/AWS's cert is monster fuckup, and that we need to understand _why_ that happened before trying anything again. [1] this makes automatic pen testing interesting because everything we scan has vulnerabilities for NFS/CIFS, FTP and TCP dns.

a012 11mo ago
Security team in most of the corporates is just a bunch of checklists markers, so for zscaler, crowdstrike or whatever they’re doing for compliance and/or certification and you can’t say no to it because it’s the company policy and who know better than “security” team?
- devdp430 11mo ago
  This is 100% true.
- lisbbb 11mo ago
  The problem I have with them is they are a no-talent crowd who enjoy pushing everyone around.

NicolaiS 11mo ago

Got acquired by a Fortune 500 and recieved new laptop. First hour I'm seeing TLS errors everywhere except the browser. They'd half-baked their internal CA rollout, so wasn't trusted properly. By day two I started validating their setup. The CA literally had a typo in the company name, not a great sign. A quick check with badssl.com showed that any self-signed(!) cert was being transparently MITM'ed and re-signed by their trusted corporate cert. Took them 40 days to fix it. Another fun side-effect of this is that devs will just turned off TLS verification, so their codebase is full of `curl -k`, `verify_mode = VERIFY_NONE`, `ServerCertificateValidationCallback = () => true`, ... Exactly the thing you want to see at a big fintech company /s

lisbbb 11mo ago
I've experienced similar. It has definitely made me less enthusiastic about working for any of those fools ever again. It's all just an exercise in mediocrity. The illiterate emails people send out are even worse--I swear that a lot of US born adults are functionally illiterate.

arianvanp 11mo ago

Complains about TLS inspection, yet fronts their website on the biggest and most widely deployed TLS introspection middle box in the world ... Why do we all disdain local TLS inspection software yet half the Internet terminates their TLS connection at Cloudflare who are most likely giving direct access to US Intelligence? It's so much worse as it's infringing on the privacy and security of billions of innocent people whilst inspection software only hurts some annoying enterprise folks. I wish we all hopped off the Cloudflare bandwagon.

phito 11mo ago
I wish so too, same for all the self-hosters using tailscale...
- kreetx 11mo ago
  These are not the same thing, the parent is confused..
- dns_snek 11mo ago
  Tailscale connections don't get terminated by a middle box, it's just end-to-end encrypted Wireguard under the hood. Cloud-hosted control panel is a risk because they could push malicious configuration changes to your clients (ACLs and new nodes if you're not using the lock feature), but they can't do it without leaving a trace like Cloudflare can.
- progbits 11mo ago
  Tailscale cannot passively observe traffic. They could inject malicious keys into your config but would be hard to mask the evidence of that.
  - treesknees 11mo ago
    Would it be hard? I thought the point of tailscale was not having to manage or concern yourself with key distribution.

apexalpha 11mo ago

I largely agree with the author. When our SOC wanted to implement TLS inspection I blocked it. Mostly because we not nearly at the security level for this, but also because it just fucks with so many things. That said, we are not a business dealing with highly sensitive data or legal responsibilities surrounding data loss prevention. If you are a business like that, say a bank or a hospital, you want to be able to block patient / customer data leaving your systems. You can do this by setting up a regex for a known format like patient numbers or bank account numbers. This requires TLS inspection obviously. Though this makes it harder to steal this data, not impossible. It does however allow the C-suite to say they did everything they could to prevent it.

apexalpha 11mo ago
Oh and the software (Netskope) was only able to decrypt our traffic in the cloud. Lmao not in a million fucking years will I upload our data to an American company in fucking plaintext.
- dcminter 11mo ago
  Netskope and the other DLP tools at my last gig would completely lock up my network connection for around 30 seconds every hour or two while maxing out 100% of a core. Fun times. The issue was still there a year after I first encountered it so I have grave doubts about the competence of those vendors. On the other hand I am sympathetic to the needs of big regulated orgs to show they're doing something to avoid data loss. It's a painful situation.

franga2000 11mo ago

I agree with the sentiment, but this part is complete bullshit: > what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously Who cares? It's not like all CAs would have to be breached, just one. CA certs are not scoped, so the moment one CA gets breached, we're all fucked. CT helps, but AFAIK it's still not enforced everywhere yet

nubinetwork 11mo ago

This is why I don't use the computers at work for anything not work related. They've been spying on us for at least 10 years.

nly 11mo ago

"If you use my (private) network you follow my rules" And I find it hard to argue with that. I've been using a VPN habitually on my phone and my (personal) laptop for a decade now. Work, home, travel. Doesn't matter. It's always on.

redrix 11mo ago
How do you find your typical daily battery life with it always on? I’ve tried this in the past and had to revert as I found it made a noticeable difference in my day-to-day. Curious to hear the experience of others.
- hexbin010 11mo ago
  Was it OpenVPN that you tried in the past? Wireguard seems much better
- letmetweakit 11mo ago
  I have the impression tailscale drains my battery on macOS and iOS, only turn it on when truly needed.
  - newdee 11mo ago
    Yeah, it most certainly does. Very noticeable on iOS. I don’t know if this is an Apple specific thing, or if it’s a similar story on Android. It’s WireGuard underneath, which is designed to not be very chatty when idle, so I’d put this down to regular back and forth with Tailscale’s control plane, relays, etc. It’s a shame really, because a huge value prop of TS is that it’s a VPN you just leave on and forget about. I hate having to toggle it when I inevitably forget to and wonder why I’m getting connection errors to private resources.
- ectospheno 11mo ago
  I have an ipv6 wireguard vpn from my iOS phone to my home network. It routes all traffic through my home isp. I use the wireguard iOS client. Battery life has been fine for me. One caveat is that background updates are disabled for almost every app.

Wowfunhappy 11mo ago

I work for a school. My traffic is not MITM'd, but the kids' traffic is, because we don't want them using their school-issued laptops to play games or go shopping, and you can't adequately block stuff if it's all encrypted.

TheChaplain 11mo ago
I can't imagine the headache as a school when parents come yelling "why did you allow my child on site XXX?!"
lousken 11mo ago
Whitelists instead of blacklists?
- Wowfunhappy 11mo ago
  I'm pretty sure we'd still need to break TLS. Domain-level just isn't granular enough.
  - lousken 11mo ago
    I still find that dumb that you even need to do that. Machines especially for schools should be able to have software policies set directly on them to limit such sites. I don't know how much chromeOS is configurable and if you can e.g. force it to only use specific network and network interface, or if a student can connect it to a different network somehow, because it would be kinda pointless otherwise.
    - Wowfunhappy 11mo ago
      The school-issued laptops are all Macbooks. To be clear I'm not in the IT department so I don't know exactly what the setup is, but I see my students using their computers. A VPN is involved, which is what made me assume they are doing TLS shenanigans—I guess I could theoretically be wrong, but it's definitely more granular than domain-level blocking, so I don't know how else it could work. The computers connect to this VPN automatically on startup. In the moments before the VPN connects, the internet does not work. > Machines especially for schools should be able to have software policies set directly on them to limit such sites. It's a good point—if you just did this client-side instead of on the network level, you wouldn't have to deal with TLS or anything. It seems clear to me that they aren't doing that (given the VPN) and it's not immediately obvious to me why.

kotaKat 11mo ago

zScaler is a load of shit, especially with some of its absolutely dumb policies like “malicious TLDs”. Because the Framework laptop site at frame.work is malicious, of course. God, I love CURLing crap from my workstation and not getting the files I needed but instead a bunch of mangled HTML telling me zScaler was going to scan what I was going to download. Bonus points that it puts me in the wrong country because I’m closer to Montreal than any American locations so half the time I’m stuck in French Canadian on the web from my New York office. Triple bonus points that I’m required to test speed at client sites and zScaler completely mangles our presentable results. Quadruple bonus points that I put in "because I feel like it" into every elevation request I make on my corporate machine and our "cyber team" has literally never looked at elevation reports to ask what the hell I'm doing...

cornonthecobra 11mo ago

> Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously? Considering that CloudFlare has managed to MitM a huge part of the internet, I'd say that probability is not just non-zero, but greater than by a worrying margin.

acdha 11mo ago
That’s not what MITM means, and also misunderstands how CAs work. Cloudflare is a concern for how many people would be affected if there was another Cloudbleed but misstating their relationship with their customers isn’t going to accomplish anything.
- JetSpiegel 11mo ago
  How is that not a MITM? Just because it's the modern day CryptoAG?
  - acdha 11mo ago
    Because it’s not an attack but rather a voluntary infrastructure choice by a company. We don’t say that Varnish is a MITM because it’s in front of my application, because it’s intentional and under my control. Misusing the term muddies the topic rather than adding clarity, and while there’s a very useful discussion about centralization or why Cloudflare’s most stringent customers might want to deploy their Keyless SSL service that discussion won’t happen if someone misuses the term.
    - JetSpiegel 11mo ago
      Yes, but Cloudflare is not under your control, that's the point.

kreetx 11mo ago

The title should say "Stop inspecting TLS", the current title reads like the TLS standard or technology is modified in a way to not work properly.

Wowfunhappy 11mo ago
For what it's worth, I knew exactly what this was going to be about before I clicked.

voidUpdate 11mo ago

I'm hoping this doesn't apply to things like Fiddler, because without the ability to see what's actually coming over the wire with a https connection, things can be a nightmare to debug sometimes

bandie91 11mo ago

personally i'm happy that i can MITM my docker when it wants to pull gigs of images the 1000th time upstream and just serve them from a local OCI cache server instead.

mirashii 11mo ago
You don't need to MITM docker for this, you can just configure your containerd or equivalent backend properly.
- baobun 11mo ago
  How about that build script cloning the same git repo or wgetting the same release bundle over and over?

dminuoso 11mo ago

One thing that has not quite been mentioned in the blog, is how much of the MITM spyware comes from very big and well known „security“ companies. You know, the ones that really know about security. X-PAN-AUTHCHECK type of security. The amount of CVEs some of the big firewall companies collect make it seem like it is a competition for the poorest security hygiene. The real problem we have is compliance theatre where someone in management forces these solutions onto their IT department just so they can check a box on their sheets and shift all responsibilities away.

YesThatTom2 11mo ago

More and more big customers (especially banks) are requiring this kind of self-inflicted-MITM attack from all their suppliers. Do you want to have customers? Get ready for zscaler! How do you propose compliance with their exfiltration protection requirements? (And “turn down $ from those customers” is not an answer)

deleted 11mo ago
[deleted]

lousken 11mo ago

As a sysadmin I also hate this. Instead, I do block stuff based on DNS requests and I also block any other DNS provider as well as malicious IPs. At this point in time, Microsoft is the bigger enemy here - some of their policies are just insane and none of this MITM will help [0][1] [0] https://www.microsoft.com/en-us/microsoft-365/roadmap?id=490064 https://www.microsoft.com/en-us/microsoft-365/roadmap?id=490... [1] https://techcommunity.microsoft.com/blog/microsoft365copilotblog/employees-can-bring-copilot-from-their-personal-microsoft-365-plans-to-work---wh/4458212 https://techcommunity.microsoft.com/blog/microsoft365copilot...

sqbic 11mo ago

What changed my mind to be in favor of TLS inspection at work environments was seeing what kind of highly confidential stuff employees might be copy-pasting to random websites, LLM assistants, cloud-based "desktop applications" and such against the approved use policies of each of these tools without giving it a second thought. TLS inspection products can intercept the paste transaction before the data leaves the company network, hitting the user with a "No you didn't! Shame on you!"-banner and notify the admins how a user just tried to paste hundreds of customers' personal information and credit card details into some snooping website, or into otherwise allowed LLM chat which still is not allowed to be used with confidential information. There can even be automations to lock the user/device out immediately if something like this is going on, be it the user or some undetected malware in the user's device attempting the intercepted action. Being able to do these kinds of very specifically targeted interceptions can prevent potentially huge disasters from happening while still allowing users more freedom in taking advantage of the huge variety of productivity tools available these days. No need to choose between completely blocking all previously unseen tools or living in fear of disastrous leaks when there are fine-grained possibilities to control what kind of information can be fed to the tools and from where. There are plenty of organizations out there where it is completely justified to enforce such limitations and monitoring in company devices. Policies can forbid personal use entirely where it is deemed necessary and legal to do so. Of course the policies and the associated enforced monitoring needs to be clearly communicated and there needs to be carefully curated configurations to control where and how TLS is or isn't intercepted so employee privacy laws and regulations aren't breached either.

iso1631 11mo ago
So deploy end point security, which sits in the kernel and can thus access the unencrypted communication
- treesknees 11mo ago
  Aren’t most TLS implementations still using things like OpenSSL in userspace? How would the kernel get access to the request?

iamcalledrob 11mo ago

This kind of TLS "man in the middle" tech is so frustrating to deal with, because it ends up breaking things. For example, I've encountered zscaler setups in the wild which close TLS connections if non-HTTP traffic is encountered. Presumably the traffic inspection fails since there is no HTTP request, and this failure path closes the socket. It's hard to say whether it's due to the customer's IT dept's config, or zscaler itself -- but as far as the customer is concerned, it's my problem.

wolfi1 11mo ago

a company, I worked for, had their own endpoint which you can easily introduce in windows, unfortunately every other tls connection which does not use the windows certificate store breaks because of that, so maven, npm et al won't work

zmmmmm 11mo ago

I'd like to know if there is any real world data about the effectiveness of TLS inspection in preventing harm. It's empirically a massive tax on any organisation that engages in any kind of software engineering or technical work. I estimate that something like 3% of all our engineering effort goes to working around the TLS inspection which breaks security on literally every piece of infrastructure we build. And it embeds all the harm that the article alludes to (and more). So it takes quite a significant balance of upside to counteract that. So is the benefit worth it? Is there data to prove it? Or is it just authoritarian IT departments drunk on power implementing this stuff? I'd love to know.

1970-01-01 11mo ago

Isn't the accepted risk of losing all the digital keys equal to the risk of someone booting up the root CA and walking away with it? I can see that happening in a junta situation. https://www.iana.org/dnssec/ceremonies https://www.iana.org/dnssec/ceremonies

acdha 11mo ago

I really wish more places would enable explicit proxies: if you have a mandate to inspect all traffic, block 443 except to proxy.megacorp.com and configure clients to use it. You lose all of the bugs and security issues caused by the security software—fun fact, Palo Alto _still_ doesn’t correctly implement TLS 1.2!—which as the author points out is basically training users to disable validation or ignore errors. This is so common I’ve seen security people with the usual certs giving advice from ChatGPT to add -k to curl calls, which is especially great when that’s baked into a container which will run on other networks. One really nice win is troubleshooting: inspection appliances break decades of performance and reliability work which groups like the Linux kernel developers have done, and they make all errors have the same symptom so the security team will now be the only people who can troubleshoot network-level issues. Explicit proxying can’t fix your network but it makes it very clear where the problem needs to be investigated.

rini17 11mo ago
Are you aware how https proxying works? Clients use CONNECT method and after that everything is opaque to the proxy. So without mitm you only know remote IP address.
- acdha 11mo ago
  There’s another difference which is what I was referring to: in both cases, your proxy has to forge the SSL certificate for the remote server but in the transparent case it also must intercept network traffic intended for the remote IP. That means that clients can’t tell whether an error or performance problem is caused by the interception layer or the remote server (sometimes Docker Hub really is down…) and it can take more work for the proxy administrator to locate logs. If you explicitly configure a proxy, the CONNECT method can trigger the same SSL forgery but because it’s explicit the client’s view is more obvious and explainable. If my browser gets an error connecting to proxy.megacorp.com I don’t spend time confirming that the remote service is working. If the outbound request fails, I’ll get a 5xx error clearly indicating that rather than having to guess at what node dropped a connection or why. This also provides another way to implement client authentication which could be useful if you have user-based access control policies. It’s not a revelation, but I think this is one of the areas where trying to do things the easy way ends up being harder once you factor in frictional support costs. Transparent proxying is trading a faster rollout for years of troubleshooting.

lisbbb 11mo ago

I'm glad someone is speaking out about this. When I first found out that the major Fortune 500 company I was working for did this, I was happy I never did any online banking or any personal finance logins using the work computer. It's obscene, quite frankly. I have many criticisms of big-corpo security and how overbearing and paranoid it has become such that developers can barely get a thing done in a lot of these big firms. Plus, most of the internal threats are embezzlers who get away with it (for awhile, at least). I did work for one place that had a Chinese national attempt to make off with the entire customer database, but he did it by burning CDs (circa 2006).

dajt 11mo ago

This is a massive pain at my work, same as I'm sure most other comments are saying. I use Platform.IO for firmware development and can't build my firmware unless I hotspot my phone. I would say that's a PIO bug unless there is a flag I don't know about, but it's exposed by this nuisance of a firewall. The devs where I am spend much of our time hotspotted to our phones with the corporate network never connected so IP goes out over the mobile network. Whenever possible we use 'do not verify server certs' flags in libs and commands which is not ideal.

parliament32 11mo ago

The author is complaining a lot about implementation pains without taking a step back and looking at why it exists in the first place. Say you work at a place that deals with credit cards. You, as a security engineer, have a mandate to stop employees from shipping CC numbers outside the org. You can educate all you want, you can have scary policies and HR buy-in, you can have all the "Anomaly detection, Zero Trust network architecture, EDR, Netflow analysis" in the world, but exactly zero of those will stop Joe Lunchbox from copy/pasting a block with a CC number in the middle into ChatGPT. You know what will? A TLS-inspecting proxy with some DLP bits and bobs. It sucks, yes. But it works, and (short of the fool's errand of trying to whitelist every site any employee needs) it's the only thing that works. And yes, I'm aware PCI DSS has additional requirements for CDEs and whatnot, but really this can apply to anything -- a local government office dealing with SSNs, a school with student ID numbers, a corporation with trade secrets.. these problems exist everywhere, and implementing PCI-like controls is often a bridge too far for unregulated industries.

dadrian 11mo ago
That is not true, you can run DLP on an endpoint directly and inside a browser directly (e.g. via an extension or direct integration hooks). You can also try to stop the situation where the CC numbers are in the clear anywhere in the first place, so that you can't copy/paste them around. What happens if someone writes the CC number down on a piece of paper?
- parliament32 11mo ago
  Endpoint DLP helps but it's not even close to bulletproof. Just for fun, if you have DLP at work, open the integrated browser in VS Code and notice how you can send protected test strings without anything chirping you. > CC numbers are in the clear anywhere in the first place Sounds great in theory, until you realize that in a large number of industries the majority of employees need access to protected data to do their jobs. Imagine telling the IRS their employees can't see/use cleartext SSNs. As for paper / mobile phones / whatever.. you're not wrong, but physical security is typically someone else's job.