5 ms·

I found the issue to be more insidious. Don’t roll your own crypto. Ok, that makes sense. Don’t implement existing crypto algorithms. Ok, arguments sound good

by wbond 10y ago

I found the issue to be more insidious. Don’t roll your own crypto. Ok, that makes sense. Don’t implement existing crypto algorithms. Ok, arguments sound good. This leaves the options of using existing libraries. Cool. So let’s go find an existing library to deal with validating an SSL certificate chain in Python. PyCrypto (https://www.dlitz.net/software/pycrypto/api/current/ https://www.dlitz.net/software/pycrypto/api/current/) - hmm, so basically the OpenSSL API, probably a minefield unless you are really up on crypto. M2Crypto (https://gitlab.com/m2crypto/m2crypto https://gitlab.com/m2crypto/m2crypto) - umm, so no real docs. The recommendation is to go read a book about network security with OpenSSL. So after reading a network security book, I should be good to wire together some OpenSSL, right? cryptography (https://cryptography.io/en/latest/ https://cryptography.io/en/latest/) - sweet, we have docs! Hmm, so for humans we have Fernet and the ability to look at X.509 certificates. But nothing about validating. Oh, but there is a stalled PR (https://github.com/pyca/cryptography/pull/2460 https://github.com/pyca/cryptography/pull/2460) from 14 months ago, to verify a certificate signatures. Which then leaves the typical programmer to – hack something together? So it is (practically) 2017, and using one of the most popular programming languages, you can’t verify an SSL certificate in a sane way without becoming journeyman cryptographer. I like crypto, but using cryptography to integrate with existing protocols/standards sucks. Can we really not have an end-user focused implementation of RFC 5280 with a few knobs to turn? Perhaps people would stop "rolling their own" crypto if there were half-decent, maintained, documented solutions out there. Maybe some day!

paulddraper 10y ago
And you used Python as the example, probably has the easiest cryptography libraries to use. C/C++, Ruby, JS, etc. are much, much worse in usability.
wolf550e 10y ago
Look at this code from urllib3 (requests bundles this code): https://github.com/shazow/urllib3/blob/master/urllib3/contrib/pyopenssl.py https://github.com/shazow/urllib3/blob/master/urllib3/contri... If you install the dependencies, requests uses the package cryptography to use openssl to validate certificates, by running that code. Note that openssl does not validate certificates the way Mozilla Firefox does, even if they use the exact same root CA bundle, because some of the validation logic is code in NSS instead of data (e.g. certificate transparency requirements). Ryan Sleevi wrote on tweeter that Red Hat are working on doing something to let applications using openssl use Firefox's full cert validation logic.
- wbond 10y ago
  (Just to note, this wasn't me looking for help in how to accomplish that task.) That said, does the proverbial "you" know all of the chain validation features that are NOT implemented by OpenSSL when using code such as in urllib3? What if you actually care more about revocation than utmost in connection performance? How does one trust a custom CA root? What if you want to verify a cert chain for something other than a TLS connection? My main point is: here we are with people needing crypto, and (it seems) no one has taken it upon themselves to write good crypto libraries for (some of) the types of tasks that are fairly common. The obvious exception being NaCl. However, that has issues with pragmatic things like distribution due to CPU optimizations, hence libsodium.
- daenney 10y ago
  > Look at this code from urllib3 (requests bundles this code): https://github.com/shazow/urllib3/blob/master/urllib3/contri.. https://github.com/shazow/urllib3/blob/master/urllib3/contri.... This only underscores/validates wbond's critique though. Apparently the advise on how to figure this stuff out isn't even through "RTFM", docs or an easy to use library, it's "go look at the dependency of this other library which uses that other dependency that you want to use on how it is done". And then copy-paste that and hopefully not get anything wrong along the way.
pvg 10y ago
Serious question: what are the common, practical situations you've run into or can think of where you need to be validating certificate chains yourself in your application code?
- wbond 10y ago
  Document signing is what pushed me down that path. It uses the same ecosystem and technologies as TLS, but cares more about revocation.
  - pvg 10y ago
    Ah I see. Seems a bit off the beaten path already, though. Were they some sort of existing documents, generated by something not easily available server side?
  - wolf550e 10y ago
    Did you require online (not cached) revocation check? That would require buying more HSMs.
    - wbond 10y ago
      No, I'm just implementing code that signs and verifies PDF digital signatures for compliance with EU regulations. I'm not dealing with HSMs, or even the PKIX infrastructure other than verifying a certificate is valid according to Adobe/ETSI and generating/checking signatures. The certificate issuers need to provide the CRLs and OCSP responders/responses.
    - wolf550e 10y ago
      Why is this downvoted? Could the downvoter please comment?
      - viraptor 10y ago
        Online revocation check doesn't require buying HSMs. HSMs also usually operate on a much lower level / aren't really connected to the internet.
PopsiclePete 10y ago
The Python eco-system is a ghetto. I don't understand why, it really is one of the most popular languages.
problems 10y ago
Well, this is a very particular case and it's only so bad because you're dealing with SSL and X509. X509 is a clusterfuck of its own. OpenSSL or OpenSSL-esque (like LibreSSL or BoringSSL), but in OpenSSL this essentially just boils down to X509_verify_cert, you really don't need to think much about the crypto. You might need to care a whole lot about X509 when dealing with loading your certificates and the complete chain though. But overall if you have a nice python wrapper that shouldn't be too bad. Not pleasant, but not too bad. We do have several quite pleasant cryptography libraries like libsodium, SJCL, etc, but none of these deal with x509.
- wbond 10y ago
  Or PKCS#12, or PKCS#8, or ECDSA (OpenSSL has determinism now, but different than RFC 6979), or ASN.1. Basically anything where you are using crypto as part of another protocol, not simply for encrypt/decrypt, sign/verify. So using crypto code will be good once we rid the world of contemporary code and move into the future where all protocols are implemented on top of poly1305, ed25519 and chacha20. Considering how long it took to get rid of SSLv2 and v3, I just think perhaps some effort should go into making good, usable solutions for crypto that needs to be used now.
- TwoBit 10y ago
  "but none of these deal with x509" Thus you have proven his point.
  - problems 10y ago
    His point was that crypto wasn't pleasant. In reality it's that x509 isn't pleasant.
guitarbill 10y ago
> Perhaps people would stop "rolling their own" crypto if there were half-decent, maintained, documented solutions out there. Maybe some day! For transport security, HTTPS or SSH. Why go lower level? Cert based stuff sucks, and cert management sucks. All of these suck harder because of backwards compatibility. Slowly, people are coming round to the fact you need to just deprecate stuff and get on with our lives. But the most sucky thing is OpenSSL. Sometimes, I think OpenSSL has actually done more damage than good. You accept a shitty library because everybody uses it so it must be secure, right? Wrong. And the API is so hostile, the docs are awful. Most things that use it or try to replace it are awful (M2Crypto, PyCrypto, PyOpenSSL, even urllib3 [0]), as if the awfulness of OpenSSL seeps into your thinking - I know it's happened to me once or twice, just interfacing with OpenSSL. Even cryptography suffers from being based on OpenSSL. Every time I have to use OpenSSL on a new project, I can't wait to see it die. [0] https://github.com/shazow/urllib3/blob/master/urllib3/util/ssl_.py https://github.com/shazow/urllib3/blob/master/urllib3/util/s...
- paulddraper 10y ago
  > For transport security, HTTPS or SSH. Why go lower level? Because not everything on this green earth uses HTTPS and SSH. Examples: (1) I want to implement a SAML consumer, and per the spec, I need to verify signatures. Crypto. (2) I want to use a client-side cookie so that users can remain authenticated in the current browser session. Crypto. (3) I want to issue a URL with a signed assertion that the owner of the content has granted permission to access it. Crypto. Asymmetric encryption is not the be-all end-all of cryptography.
- wbond 10y ago
  I'm not going lower level – in fact, I'm not even dealing with a communication protocol. Certificates are also used for document and executable signing.
  - guitarbill 10y ago
    Yeah, fair enough - it was more of a general assertion that it's easy to forget that high-level solutions exist. It's NIH. Does using certs still suck though? I get that it's a hard problem, but cert revocation and distribution, that's really hard even for OSes.

by wbond 10y ago

paulddraper 10y ago

And you used Python as the example, probably has the easiest cryptography libraries to use. C/C++, Ruby, JS, etc. are much, much worse in usability.

wolf550e 10y ago

Look at this code from urllib3 (requests bundles this code): https://github.com/shazow/urllib3/blob/master/urllib3/contrib/pyopenssl.py https://github.com/shazow/urllib3/blob/master/urllib3/contri... If you install the dependencies, requests uses the package cryptography to use openssl to validate certificates, by running that code. Note that openssl does not validate certificates the way Mozilla Firefox does, even if they use the exact same root CA bundle, because some of the validation logic is code in NSS instead of data (e.g. certificate transparency requirements). Ryan Sleevi wrote on tweeter that Red Hat are working on doing something to let applications using openssl use Firefox's full cert validation logic.

wbond 10y ago
(Just to note, this wasn't me looking for help in how to accomplish that task.) That said, does the proverbial "you" know all of the chain validation features that are NOT implemented by OpenSSL when using code such as in urllib3? What if you actually care more about revocation than utmost in connection performance? How does one trust a custom CA root? What if you want to verify a cert chain for something other than a TLS connection? My main point is: here we are with people needing crypto, and (it seems) no one has taken it upon themselves to write good crypto libraries for (some of) the types of tasks that are fairly common. The obvious exception being NaCl. However, that has issues with pragmatic things like distribution due to CPU optimizations, hence libsodium.
daenney 10y ago
> Look at this code from urllib3 (requests bundles this code): https://github.com/shazow/urllib3/blob/master/urllib3/contri.. https://github.com/shazow/urllib3/blob/master/urllib3/contri.... This only underscores/validates wbond's critique though. Apparently the advise on how to figure this stuff out isn't even through "RTFM", docs or an easy to use library, it's "go look at the dependency of this other library which uses that other dependency that you want to use on how it is done". And then copy-paste that and hopefully not get anything wrong along the way.

pvg 10y ago

Serious question: what are the common, practical situations you've run into or can think of where you need to be validating certificate chains yourself in your application code?

wbond 10y ago
Document signing is what pushed me down that path. It uses the same ecosystem and technologies as TLS, but cares more about revocation.
- pvg 10y ago
  Ah I see. Seems a bit off the beaten path already, though. Were they some sort of existing documents, generated by something not easily available server side?
- wolf550e 10y ago
  Did you require online (not cached) revocation check? That would require buying more HSMs.
  - wbond 10y ago
    No, I'm just implementing code that signs and verifies PDF digital signatures for compliance with EU regulations. I'm not dealing with HSMs, or even the PKIX infrastructure other than verifying a certificate is valid according to Adobe/ETSI and generating/checking signatures. The certificate issuers need to provide the CRLs and OCSP responders/responses.
  - wolf550e 10y ago
    Why is this downvoted? Could the downvoter please comment?
    - viraptor 10y ago
      Online revocation check doesn't require buying HSMs. HSMs also usually operate on a much lower level / aren't really connected to the internet.

PopsiclePete 10y ago

The Python eco-system is a ghetto. I don't understand why, it really is one of the most popular languages.

problems 10y ago

Well, this is a very particular case and it's only so bad because you're dealing with SSL and X509. X509 is a clusterfuck of its own. OpenSSL or OpenSSL-esque (like LibreSSL or BoringSSL), but in OpenSSL this essentially just boils down to X509_verify_cert, you really don't need to think much about the crypto. You might need to care a whole lot about X509 when dealing with loading your certificates and the complete chain though. But overall if you have a nice python wrapper that shouldn't be too bad. Not pleasant, but not too bad. We do have several quite pleasant cryptography libraries like libsodium, SJCL, etc, but none of these deal with x509.

wbond 10y ago
Or PKCS#12, or PKCS#8, or ECDSA (OpenSSL has determinism now, but different than RFC 6979), or ASN.1. Basically anything where you are using crypto as part of another protocol, not simply for encrypt/decrypt, sign/verify. So using crypto code will be good once we rid the world of contemporary code and move into the future where all protocols are implemented on top of poly1305, ed25519 and chacha20. Considering how long it took to get rid of SSLv2 and v3, I just think perhaps some effort should go into making good, usable solutions for crypto that needs to be used now.
TwoBit 10y ago
"but none of these deal with x509" Thus you have proven his point.
- problems 10y ago
  His point was that crypto wasn't pleasant. In reality it's that x509 isn't pleasant.

guitarbill 10y ago

> Perhaps people would stop "rolling their own" crypto if there were half-decent, maintained, documented solutions out there. Maybe some day! For transport security, HTTPS or SSH. Why go lower level? Cert based stuff sucks, and cert management sucks. All of these suck harder because of backwards compatibility. Slowly, people are coming round to the fact you need to just deprecate stuff and get on with our lives. But the most sucky thing is OpenSSL. Sometimes, I think OpenSSL has actually done more damage than good. You accept a shitty library because everybody uses it so it must be secure, right? Wrong. And the API is so hostile, the docs are awful. Most things that use it or try to replace it are awful (M2Crypto, PyCrypto, PyOpenSSL, even urllib3 [0]), as if the awfulness of OpenSSL seeps into your thinking - I know it's happened to me once or twice, just interfacing with OpenSSL. Even cryptography suffers from being based on OpenSSL. Every time I have to use OpenSSL on a new project, I can't wait to see it die. [0] https://github.com/shazow/urllib3/blob/master/urllib3/util/ssl_.py https://github.com/shazow/urllib3/blob/master/urllib3/util/s...

paulddraper 10y ago
> For transport security, HTTPS or SSH. Why go lower level? Because not everything on this green earth uses HTTPS and SSH. Examples: (1) I want to implement a SAML consumer, and per the spec, I need to verify signatures. Crypto. (2) I want to use a client-side cookie so that users can remain authenticated in the current browser session. Crypto. (3) I want to issue a URL with a signed assertion that the owner of the content has granted permission to access it. Crypto. Asymmetric encryption is not the be-all end-all of cryptography.
wbond 10y ago
I'm not going lower level – in fact, I'm not even dealing with a communication protocol. Certificates are also used for document and executable signing.
- guitarbill 10y ago
  Yeah, fair enough - it was more of a general assertion that it's easy to forget that high-level solutions exist. It's NIH. Does using certs still suck though? I get that it's a hard problem, but cert revocation and distribution, that's really hard even for OSes.