3 ms·

A large company I worked at a few years ago had an internal Python channel in Teams for coding support. So many questions were about SSL issues, people would j

A large company I worked at a few years ago had an internal Python channel in Teams for coding support. So many questions were about SSL issues, people would just ask how to disable errors/warnings from not having the correct certificate chain installed. It was insane how many "helpful" people would assist in turning them off instead of simply fixing the problem. I started showing people the correct way to fix the issue and also created documentation to install the internal certificate server on our Ubuntu servers (I think they had it working on some of the RHEL machines). I was a contractor so I received an $80 bonus for my efforts.

gertop 2y ago
> instead of simply fixing the problem. Your view is probably skewed because you were the expert but I can assure you that fixing certificate issues is not a simple process for the vast majority of us, especially 15 years ago. See the sibling comment by lucb1e for a description of what the typical experience is like when trying to solve such issue.
ethbr1 2y ago
> Python channel in Teams for coding support. So many questions were about SSL issues I learned the other day that Python doesn't support AIA chasing natively. https://bugs.python.org/issue18617 https://bugs.python.org/issue18617 (Certs configured that way are technically incomplete, but because other browsers etc. handle it, it's now a "python breaks for certificates that work for other pieces of software" situation)
- anttihaapala 2y ago
  The issue was migrated to github so more up-to-date discussion is in https://github.com/python/cpython/issues/62817 https://github.com/python/cpython/issues/62817
  - consp 2y ago
    This discussion is just "do it because some browsers do it" without any reasoning why (or why not) you should do it. Firefox approach is i guess the best compromise between user annoyance and developer annoyance but it's still a compromise against proper TLS.
- richm44 2y ago
  Downloading things from the AIA fields would mean triggering HTTP/HTTPS requests to an untrusted URL from a certificate you haven't verified - not a good idea. What firefox does is cache intermediates that it has seen elsewhere, the windows TLS stack can fetch additional certs from windows update on-demand (and actually starts with only a small bundle of trusted roots). There is no good solution for incomplete chains other than getting the sites fixed (or using a provider like cloudflare that solves it for them).
- zerocrates 2y ago
  I don't think I've seen anything but a browser ever do this, fixing an incomplete chain. curl, wget, several different programming languages, everything just fails to verify. I can understand why it wouldn't be supported, but you also see why users and developers experience this as just "SSL/TLS just gives you these weird errors sometimes" and pass around solutions to turn off verification.
TechDebtDevin 2y ago
You'd be surprised how many companies with insanely valuable IP (especially in the startup space) who do not use vaults/secret managers and store keys in plain text files. Its pretty astonishing tbh.
- xvector 2y ago
  Even at large companies. Secrets management was not even being done across large swaths of FAANG companies until ~2020. I know some people that made a very lucrative career out of enabling secrets at these orgs from 2010-2020.
TeMPOraL 2y ago
> instead of simply fixing the problem. No such thing when certificates are involved. You basically have two options to do it "correctly": 1) Walk a path of broken glass and razorblades, on your naked knees, through the depths of hell, trying to get a complex set of ancient tools and policies that no one truly understands to work together. One misstep, the whole thing seizes up, and good luck debugging or fixing it across organizational boundaries. 2) Throw in the towel and expose the insides of your org, and everyone you come into contact with, on the public Internet, so you can leverage "Internet-standard" tools and practices. One of the fundamental issues is that doing SSL properly breaks a basic engineering assumption of locality/isolation. That is, if I'm making a tool that talks to another tool (that may or may not be made by me too) directly, I should only care about the two tools and the link between them. Not the goddamn public Internet. Alas, setting SSL means either entangling your tool with the corporate universe, or replicating a facsimile of the entire world locally, just so nothing in the stack starts whining about CAs, or that self-signed certs smell like poop, or something. Like seriously. You make a dumb internal tool for yourself, with a web interface. You figure you want to do HTTPS because browsers whine (o. Apparently the correct way of doing this is... to buy a domain and get a cert from LetsEncrypt. WTF. The whole philosophy around certificates is not designed to facilitate development. And guess what, I too sometimes get requests to give ability for a tool to skip some checks to make product testing possible, and it turns out that the whole communication stack already has flags for exactly that, for exactly that reason. EDIT: Imagine an arm broke off your coat hanger. You figure you'll take a metal bracket and two screws and fix it right there. But as you try, your power drill refuses to work and flashes some error about "insecure environment". You go on-line, and everyone tells you you need to go to the city council and register the drill and the coat hanger on a free Let's Construct build permit. This is how dealing with SSL "correctly" feels.
- arccy 2y ago
  the network is never secure, that's why there's all this stuff going on about "zero trust"
  - ninkendo 2y ago
    Please explain to me about how the “network” between my browser and my kubernetes dev installation on the same computer is insecure.

gertop 2y ago

> instead of simply fixing the problem. Your view is probably skewed because you were the expert but I can assure you that fixing certificate issues is not a simple process for the vast majority of us, especially 15 years ago. See the sibling comment by lucb1e for a description of what the typical experience is like when trying to solve such issue.

ethbr1 2y ago

> Python channel in Teams for coding support. So many questions were about SSL issues I learned the other day that Python doesn't support AIA chasing natively. https://bugs.python.org/issue18617 https://bugs.python.org/issue18617 (Certs configured that way are technically incomplete, but because other browsers etc. handle it, it's now a "python breaks for certificates that work for other pieces of software" situation)

anttihaapala 2y ago
The issue was migrated to github so more up-to-date discussion is in https://github.com/python/cpython/issues/62817 https://github.com/python/cpython/issues/62817
- consp 2y ago
  This discussion is just "do it because some browsers do it" without any reasoning why (or why not) you should do it. Firefox approach is i guess the best compromise between user annoyance and developer annoyance but it's still a compromise against proper TLS.
richm44 2y ago
Downloading things from the AIA fields would mean triggering HTTP/HTTPS requests to an untrusted URL from a certificate you haven't verified - not a good idea. What firefox does is cache intermediates that it has seen elsewhere, the windows TLS stack can fetch additional certs from windows update on-demand (and actually starts with only a small bundle of trusted roots). There is no good solution for incomplete chains other than getting the sites fixed (or using a provider like cloudflare that solves it for them).
zerocrates 2y ago
I don't think I've seen anything but a browser ever do this, fixing an incomplete chain. curl, wget, several different programming languages, everything just fails to verify. I can understand why it wouldn't be supported, but you also see why users and developers experience this as just "SSL/TLS just gives you these weird errors sometimes" and pass around solutions to turn off verification.

TechDebtDevin 2y ago

You'd be surprised how many companies with insanely valuable IP (especially in the startup space) who do not use vaults/secret managers and store keys in plain text files. Its pretty astonishing tbh.

xvector 2y ago
Even at large companies. Secrets management was not even being done across large swaths of FAANG companies until ~2020. I know some people that made a very lucrative career out of enabling secrets at these orgs from 2010-2020.

TeMPOraL 2y ago

> instead of simply fixing the problem. No such thing when certificates are involved. You basically have two options to do it "correctly": 1) Walk a path of broken glass and razorblades, on your naked knees, through the depths of hell, trying to get a complex set of ancient tools and policies that no one truly understands to work together. One misstep, the whole thing seizes up, and good luck debugging or fixing it across organizational boundaries. 2) Throw in the towel and expose the insides of your org, and everyone you come into contact with, on the public Internet, so you can leverage "Internet-standard" tools and practices. One of the fundamental issues is that doing SSL properly breaks a basic engineering assumption of locality/isolation. That is, if I'm making a tool that talks to another tool (that may or may not be made by me too) directly, I should only care about the two tools and the link between them. Not the goddamn public Internet. Alas, setting SSL means either entangling your tool with the corporate universe, or replicating a facsimile of the entire world locally, just so nothing in the stack starts whining about CAs, or that self-signed certs smell like poop, or something. Like seriously. You make a dumb internal tool for yourself, with a web interface. You figure you want to do HTTPS because browsers whine (o. Apparently the correct way of doing this is... to buy a domain and get a cert from LetsEncrypt. WTF. The whole philosophy around certificates is not designed to facilitate development. And guess what, I too sometimes get requests to give ability for a tool to skip some checks to make product testing possible, and it turns out that the whole communication stack already has flags for exactly that, for exactly that reason. EDIT: Imagine an arm broke off your coat hanger. You figure you'll take a metal bracket and two screws and fix it right there. But as you try, your power drill refuses to work and flashes some error about "insecure environment". You go on-line, and everyone tells you you need to go to the city council and register the drill and the coat hanger on a free Let's Construct build permit. This is how dealing with SSL "correctly" feels.

arccy 2y ago
the network is never secure, that's why there's all this stuff going on about "zero trust"
- ninkendo 2y ago
  Please explain to me about how the “network” between my browser and my kubernetes dev installation on the same computer is insecure.