6 ms·

Do you check HTTPS certificates in your API clients?

114 points by va1en0k 14y ago

jokull 14y ago
NGiNX reverse proxy can take care of this and does so securely, am I right?
- va1en0k 14y ago
  Probably they can. However, I don't know anything about using nginx that way. If you give me a link or something like that, I'll happily add it to my article. Or I'll just investigate later.
  - jokull 14y ago
    Yeah my bad I read your article assuming you were an API provider not a consumer. In the case of consuming API’s I suggest python-requests, an excellent library that is extremely well maintained.
- rfugger 14y ago
  No, nginx is on the server side. The article is about client-side.
  - va1en0k 14y ago
    You can use nginx as a proxy on your side for any APIs. Just proxy_pass it to the API, and listen only on the local side. It can provide caching and other stuff that way.
    - obtu 14y ago
      Yeah, but that's proxy, not reverse proxy. It's possible to do something fiddler-like using apache or nginx to mitm oneself and dump everything that goes through.
- moonboots 14y ago
  Nginx ssl reverse proxy does not verify certificates. I tested by reverse proxying to an ssl server with a self signed certificate. Nginx did not log any warnings or complain.
josephcooney 14y ago
If you want to check any 3rd party apps or frameworks that you use you can set up a proxy like Fiddler, configure it to do a MITM attack on you, and see if the client/API blows up with an error or just keeps on truckin'.
- va1en0k 14y ago
  better than /etc/hosts approach, yeah
delinka 14y ago
In this age of high-level languages, why do I still have to worry about this? I don't mean 'security' I mean 'managing certificates.' My local framework/API should complain if I don't have a trusted root and should then make it dead simple to provide that root.
- va1en0k 14y ago
  Exactly. Security is something that should just work - arguably, the first thing with such requirement. Even though it simply can't just work in many cases, it should be as close to just-working as it can.
- TeHCrAzY 14y ago
  That's exactly how it works in C# (.NET in general), and I would take a fairly large bet that's exactly the same in Java. It seems like a regression (when thinking on terms of lower level --> higher level language movement).
- rfugger 14y ago
  Python does have some certificate-checking machinery: http://docs.python.org/library/ssl.html#functions-constants-and-exceptions http://docs.python.org/library/ssl.html#functions-constants-... http://docs.python.org/library/ssl.html#ssl-certificates http://docs.python.org/library/ssl.html#ssl-certificates However, this isn't exposed in the higher-level httplib.HTTPSConnection class for some reason. I'd bet it's not too hard to write your own subclass to handle it though.
  - natrius 14y ago
    If you're using Python for HTTP requests, you should be using Requests, which checks SSL certificates by default (as far as I can tell). http://docs.python-requests.org/en/latest/ http://docs.python-requests.org/en/latest/
    - voltagex_ 14y ago
      Yeah, I posted this below. It's mentioned in the advanced section. Why checking your SSL certs is an advanced topic I don't know.
SlipperySlope 14y ago
Java HTTPS server code allows the developer to configure the X.509 trust manager. My API uses Java for both client and server and transmits JSON messages over secure web sockets which is built over HTTPS. Works great. I authenticate on both client and server, and am just now developing an iOS Objective C client that runs wss.
jspaur 14y ago
i know in the the world of .NET (atleast on Windows) it'll automatically check using the cert manager. Anyone know how this might work on platforms such as Java? I'd assume the local VM would need to have some OS specific plumbing in place.
- va1en0k 14y ago
  many servers are deployed using Linux which doesn't have any cert manager (see citation in the blog)
  - byroot 14y ago
    But most distribution provide a list of trusted root CA. For exemple on Debian: /etc/ssl/certs/
  - jspaur 14y ago
    what about with platforms such as Android? iOS? etc.
    - salgernon 14y ago
      iOS and OSX both use the system store represented by the "system" keychain. (On the desktop, individual users can also have keychain a with trusted roots.). Apple keeps their root store up to date via software updates and automatic OCSP checks. NSURLConnection, the higher level resource API, will by default require a valid certificate chain, but provides for explicitly allowing an insecure connection as part of its authentication callbacks. (This is an improvement over previous versions (leopard and before) where you had to explicitly specify hostnames that should be considered safe.) So, certificate validation is treated, at the API level, like any other sort of authentication challenge. You can also provide a client certificate using the same mechanism, if requested by the server. Using the lower level CFHTTP stream API, you can only fail the connection and re attempt it after disabling cert checking.
- stonemetal 14y ago
  The JDK comes with a program called key tool. You use key tool to build a "trust store". This is basically a collection of certs you trust. There is also a "key store" which contains your cert and its private key. Then when you run your app you must specify what key store and trust store to use.
javert 14y ago
So presumably, decent high-level languages (like Python) do use an encrypted connection - they just don't ensure the right person is on the other end of that connection?
- va1en0k 14y ago
  yep however, encrypted connection without this checks allows, for example, Man-In-The-Middle attack
tav 14y ago
When you validate server certificates from HTTPS clients, please be sure to use the right set of root certs. Mozilla maintain a decent list of these [1], but it's not in the PEM format that most HTTPS client libraries expect, e.g. Python's ssl.wrap_socket(sock, ca_certs="certs.pem"). Mozilla's list also includes distrusted certificates, so you need to be careful to leave them out when generating the PEM-encoded format. In fact, I'd strongly recommend using Adam Langley's excellent extract-nss-root-certs tool [2] which takes care of the subtle details for you. And, if you are willing to trust me, you can download my pre-generated PEM-encoded cacerts file from a month or so ago [3]. [1] https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1 https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw... [2] https://github.com/agl/extract-nss-root-certs https://github.com/agl/extract-nss-root-certs [3] https://github.com/downloads/tav/ampify/distfile.cacerts-2012.06.28.tar.bz2 https://github.com/downloads/tav/ampify/distfile.cacerts-201...
- va1en0k 14y ago
  I'd actually recommend just get the exact certificates you're going to work with. For example, if I'm using GitHub's API, it makes sense not to check all the chain, but just keep the GitHub's original cert.
  - rfugger 14y ago
    That's great until it changes.
  - tav 14y ago
    Disclaimer: I don't particularly agree with the Certificate Authority mechanism that we currently use with TLS. However, given that it's what we currently have, I'd strongly advice taking advantage of the security that it provides. Requiring API client library authors to ship certs will make for poor security. Not only do certificates expire, they also get compromised. It would be easy to conduct MITM attacks using revoked certs and API client library users would be none-the-wiser. Instead, it should be the responsibility of HTTPS client libraries to use the latest cacerts data and support features like OCSP [1] for validating certificate revocations, etc. [1] http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol http://en.wikipedia.org/wiki/Online_Certificate_Status_Proto...
pc 14y ago
Getting this right was a big pain when we were implementing the various Stripe client libraries. We had to resort to hacks even in mainstream languages.
- va1en0k 14y ago
  Well, having to use hacks for that is just plain wrong. I think we can consider this as bugs in these mainstream languages. Probably you could file them?
voltagex_ 14y ago
The requests library in Python will check your certs. docs.python-requests.org/en/latest/user/advanced/#advanced
- va1en0k 14y ago
  Why the hell is that advanced? Nice to know, though. Thanks!
  - natrius 14y ago
    Changing which certificates are used and whether to check is advanced. The right thing is the default.
- polychrome 14y ago
  This will take you straight there: http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification http://docs.python-requests.org/en/latest/user/advanced/#ssl...
  - voltagex_ 14y ago
    Thanks. Did I miss the TOC or was that a hidden anchor?
    - forsaken 14y ago
      Most sphinx documentation has a permalink if you hover over the title text. It will show up in this instance to the right.
      - polychrome 14y ago
        Ahh. I couldn't figure that out to be honest, so I just viewed the source.
deleted 14y ago
[deleted]
nl 14y ago
Java uses a keystore (actually - being Java - you can set a property to tell it which keystore to use). Importing a new certificate is documented here: http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Security10.html http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Security10.htm... The verification of the certificate is delegated to a TrustManager[1]. Certificate paths can be checked[2] and the TrustManager is flexible enough to support a large variety of verification scenarios[3]. [1] http://docs.oracle.com/javase/1.4.2/docs/api/javax/net/ssl/TrustManager.html http://docs.oracle.com/javase/1.4.2/docs/api/javax/net/ssl/T... [2] http://www.exampledepot.com/egs/java.security.cert/ValidCertPath.html http://www.exampledepot.com/egs/java.security.cert/ValidCert... [3] http://stackoverflow.com/questions/11681474/how-can-i-trust-a-specific-self-sign-certificates-not-trust-all?lq=1 http://stackoverflow.com/questions/11681474/how-can-i-trust-...
nivloc 14y ago
This is default behaviour in Ruby, and as mentioned in the article, one can set up a custom store. With all the languages that support this by default, I'm surprised Python doesn't.
stanleydrew 14y ago
The twilio-ruby library does this by packaging a set of root ca certs. There is a configuration option to override this with your own set when you construct the client.
st3fan 14y ago
Validating certificates is a good thing and everybody should do it. That said ... it really only tells you that a certificate is 'sound'. It by no means tells you with 100% confidence that you are talking to the right party. SSL/TLS is still pretty fragile.
polychrome 14y ago
Great Article! Looks like as of Curl 7.10 PHP now checks this by default: CURLOPT_SSL_VERIFYPEER CURLOPT_SSL_VERIFYHOST http://php.net/manual/en/function.curl-setopt.php http://php.net/manual/en/function.curl-setopt.php
Spooky23 14y ago
The problem with doing this is that many/most enterprise IT shops use security devices to proxy SSL traffic, altering the cert that the client sees. So unless you really understand you app's deployment scenario well, you should proceed with caution. Also note that 2% of enterprise IT people understand SSL at all, (and the devices that do the MITM part are usually controlled by the security dept) so troubleshooting will be close to impossible.
- tptacek 14y ago
  Those proxies publish their fake CA=YES certs, so you can just add them to your root and everything will validate.
celere 14y ago
Python: Just check the certificates with Python Requests (or take one of 20 lines urllib solutions): >>> requests.get('https://exmaple.com https://exmaple.com, verify=True) requests.exceptions.SSLError: hostname 'exmaple.com' doesn't match either of '*.exmaple.org', 'exmaple.org'
__alexs 14y ago
To make it worse in many languages there not built in OCSP or CRL facilities to go with their standard TLS wrappers. e.g. The best you get in Python is checking against a CA list. So even if you do go to the trouble to turn on CA verification yourself you still accept known bad certificates.
polychrome 14y ago
Node.js seems to omit using a certificate as well. From the documentation: * cert: Public x509 certificate to use. Default null. [1] http://nodejs.org/docs/latest/api/https.html#https_https_request_options_callback http://nodejs.org/docs/latest/api/https.html#https_https_req...

114 points by va1en0k 14y ago

jokull 14y ago

NGiNX reverse proxy can take care of this and does so securely, am I right?

va1en0k 14y ago
Probably they can. However, I don't know anything about using nginx that way. If you give me a link or something like that, I'll happily add it to my article. Or I'll just investigate later.
- jokull 14y ago
  Yeah my bad I read your article assuming you were an API provider not a consumer. In the case of consuming API’s I suggest python-requests, an excellent library that is extremely well maintained.
rfugger 14y ago
No, nginx is on the server side. The article is about client-side.
- va1en0k 14y ago
  You can use nginx as a proxy on your side for any APIs. Just proxy_pass it to the API, and listen only on the local side. It can provide caching and other stuff that way.
  - obtu 14y ago
    Yeah, but that's proxy, not reverse proxy. It's possible to do something fiddler-like using apache or nginx to mitm oneself and dump everything that goes through.
moonboots 14y ago
Nginx ssl reverse proxy does not verify certificates. I tested by reverse proxying to an ssl server with a self signed certificate. Nginx did not log any warnings or complain.

josephcooney 14y ago

If you want to check any 3rd party apps or frameworks that you use you can set up a proxy like Fiddler, configure it to do a MITM attack on you, and see if the client/API blows up with an error or just keeps on truckin'.

va1en0k 14y ago
better than /etc/hosts approach, yeah

delinka 14y ago

In this age of high-level languages, why do I still have to worry about this? I don't mean 'security' I mean 'managing certificates.' My local framework/API should complain if I don't have a trusted root and should then make it dead simple to provide that root.

va1en0k 14y ago
Exactly. Security is something that should just work - arguably, the first thing with such requirement. Even though it simply can't just work in many cases, it should be as close to just-working as it can.
TeHCrAzY 14y ago
That's exactly how it works in C# (.NET in general), and I would take a fairly large bet that's exactly the same in Java. It seems like a regression (when thinking on terms of lower level --> higher level language movement).
rfugger 14y ago
Python does have some certificate-checking machinery: http://docs.python.org/library/ssl.html#functions-constants-and-exceptions http://docs.python.org/library/ssl.html#functions-constants-... http://docs.python.org/library/ssl.html#ssl-certificates http://docs.python.org/library/ssl.html#ssl-certificates However, this isn't exposed in the higher-level httplib.HTTPSConnection class for some reason. I'd bet it's not too hard to write your own subclass to handle it though.
- natrius 14y ago
  If you're using Python for HTTP requests, you should be using Requests, which checks SSL certificates by default (as far as I can tell). http://docs.python-requests.org/en/latest/ http://docs.python-requests.org/en/latest/
  - voltagex_ 14y ago
    Yeah, I posted this below. It's mentioned in the advanced section. Why checking your SSL certs is an advanced topic I don't know.

SlipperySlope 14y ago

Java HTTPS server code allows the developer to configure the X.509 trust manager. My API uses Java for both client and server and transmits JSON messages over secure web sockets which is built over HTTPS. Works great. I authenticate on both client and server, and am just now developing an iOS Objective C client that runs wss.

jspaur 14y ago

i know in the the world of .NET (atleast on Windows) it'll automatically check using the cert manager. Anyone know how this might work on platforms such as Java? I'd assume the local VM would need to have some OS specific plumbing in place.

va1en0k 14y ago
many servers are deployed using Linux which doesn't have any cert manager (see citation in the blog)
- byroot 14y ago
  But most distribution provide a list of trusted root CA. For exemple on Debian: /etc/ssl/certs/
- jspaur 14y ago
  what about with platforms such as Android? iOS? etc.
  - salgernon 14y ago
    iOS and OSX both use the system store represented by the "system" keychain. (On the desktop, individual users can also have keychain a with trusted roots.). Apple keeps their root store up to date via software updates and automatic OCSP checks. NSURLConnection, the higher level resource API, will by default require a valid certificate chain, but provides for explicitly allowing an insecure connection as part of its authentication callbacks. (This is an improvement over previous versions (leopard and before) where you had to explicitly specify hostnames that should be considered safe.) So, certificate validation is treated, at the API level, like any other sort of authentication challenge. You can also provide a client certificate using the same mechanism, if requested by the server. Using the lower level CFHTTP stream API, you can only fail the connection and re attempt it after disabling cert checking.
stonemetal 14y ago
The JDK comes with a program called key tool. You use key tool to build a "trust store". This is basically a collection of certs you trust. There is also a "key store" which contains your cert and its private key. Then when you run your app you must specify what key store and trust store to use.

javert 14y ago

So presumably, decent high-level languages (like Python) do use an encrypted connection - they just don't ensure the right person is on the other end of that connection?

va1en0k 14y ago
yep however, encrypted connection without this checks allows, for example, Man-In-The-Middle attack

tav 14y ago

When you validate server certificates from HTTPS clients, please be sure to use the right set of root certs. Mozilla maintain a decent list of these [1], but it's not in the PEM format that most HTTPS client libraries expect, e.g. Python's ssl.wrap_socket(sock, ca_certs="certs.pem"). Mozilla's list also includes distrusted certificates, so you need to be careful to leave them out when generating the PEM-encoded format. In fact, I'd strongly recommend using Adam Langley's excellent extract-nss-root-certs tool [2] which takes care of the subtle details for you. And, if you are willing to trust me, you can download my pre-generated PEM-encoded cacerts file from a month or so ago [3]. [1] https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1 https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw... [2] https://github.com/agl/extract-nss-root-certs https://github.com/agl/extract-nss-root-certs [3] https://github.com/downloads/tav/ampify/distfile.cacerts-2012.06.28.tar.bz2 https://github.com/downloads/tav/ampify/distfile.cacerts-201...

va1en0k 14y ago
I'd actually recommend just get the exact certificates you're going to work with. For example, if I'm using GitHub's API, it makes sense not to check all the chain, but just keep the GitHub's original cert.
- rfugger 14y ago
  That's great until it changes.
- tav 14y ago
  Disclaimer: I don't particularly agree with the Certificate Authority mechanism that we currently use with TLS. However, given that it's what we currently have, I'd strongly advice taking advantage of the security that it provides. Requiring API client library authors to ship certs will make for poor security. Not only do certificates expire, they also get compromised. It would be easy to conduct MITM attacks using revoked certs and API client library users would be none-the-wiser. Instead, it should be the responsibility of HTTPS client libraries to use the latest cacerts data and support features like OCSP [1] for validating certificate revocations, etc. [1] http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol http://en.wikipedia.org/wiki/Online_Certificate_Status_Proto...

pc 14y ago

Getting this right was a big pain when we were implementing the various Stripe client libraries. We had to resort to hacks even in mainstream languages.

va1en0k 14y ago
Well, having to use hacks for that is just plain wrong. I think we can consider this as bugs in these mainstream languages. Probably you could file them?

voltagex_ 14y ago

The requests library in Python will check your certs. docs.python-requests.org/en/latest/user/advanced/#advanced

va1en0k 14y ago
Why the hell is that advanced? Nice to know, though. Thanks!
- natrius 14y ago
  Changing which certificates are used and whether to check is advanced. The right thing is the default.
polychrome 14y ago
This will take you straight there: http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification http://docs.python-requests.org/en/latest/user/advanced/#ssl...
- voltagex_ 14y ago
  Thanks. Did I miss the TOC or was that a hidden anchor?
  - forsaken 14y ago
    Most sphinx documentation has a permalink if you hover over the title text. It will show up in this instance to the right.
    - polychrome 14y ago
      Ahh. I couldn't figure that out to be honest, so I just viewed the source.

deleted 14y ago

[deleted]

nl 14y ago

Java uses a keystore (actually - being Java - you can set a property to tell it which keystore to use). Importing a new certificate is documented here: http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Security10.html http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Security10.htm... The verification of the certificate is delegated to a TrustManager[1]. Certificate paths can be checked[2] and the TrustManager is flexible enough to support a large variety of verification scenarios[3]. [1] http://docs.oracle.com/javase/1.4.2/docs/api/javax/net/ssl/TrustManager.html http://docs.oracle.com/javase/1.4.2/docs/api/javax/net/ssl/T... [2] http://www.exampledepot.com/egs/java.security.cert/ValidCertPath.html http://www.exampledepot.com/egs/java.security.cert/ValidCert... [3] http://stackoverflow.com/questions/11681474/how-can-i-trust-a-specific-self-sign-certificates-not-trust-all?lq=1 http://stackoverflow.com/questions/11681474/how-can-i-trust-...

nivloc 14y ago

This is default behaviour in Ruby, and as mentioned in the article, one can set up a custom store. With all the languages that support this by default, I'm surprised Python doesn't.

stanleydrew 14y ago

The twilio-ruby library does this by packaging a set of root ca certs. There is a configuration option to override this with your own set when you construct the client.

st3fan 14y ago

Validating certificates is a good thing and everybody should do it. That said ... it really only tells you that a certificate is 'sound'. It by no means tells you with 100% confidence that you are talking to the right party. SSL/TLS is still pretty fragile.

polychrome 14y ago

Great Article! Looks like as of Curl 7.10 PHP now checks this by default: CURLOPT_SSL_VERIFYPEER CURLOPT_SSL_VERIFYHOST http://php.net/manual/en/function.curl-setopt.php http://php.net/manual/en/function.curl-setopt.php

Spooky23 14y ago

The problem with doing this is that many/most enterprise IT shops use security devices to proxy SSL traffic, altering the cert that the client sees. So unless you really understand you app's deployment scenario well, you should proceed with caution. Also note that 2% of enterprise IT people understand SSL at all, (and the devices that do the MITM part are usually controlled by the security dept) so troubleshooting will be close to impossible.

tptacek 14y ago
Those proxies publish their fake CA=YES certs, so you can just add them to your root and everything will validate.

celere 14y ago

Python: Just check the certificates with Python Requests (or take one of 20 lines urllib solutions): >>> requests.get('https://exmaple.com https://exmaple.com, verify=True) requests.exceptions.SSLError: hostname 'exmaple.com' doesn't match either of '*.exmaple.org', 'exmaple.org'

__alexs 14y ago

To make it worse in many languages there not built in OCSP or CRL facilities to go with their standard TLS wrappers. e.g. The best you get in Python is checking against a CA list. So even if you do go to the trouble to turn on CA verification yourself you still accept known bad certificates.

polychrome 14y ago

Node.js seems to omit using a certificate as well. From the documentation: * cert: Public x509 certificate to use. Default null. [1] http://nodejs.org/docs/latest/api/https.html#https_https_request_options_callback http://nodejs.org/docs/latest/api/https.html#https_https_req...