6 ms·

The US government has basically declared "HTTPS/TLS Interception Considered Harmful". This is going to be interesting as all the major security load blanacer/ap

The US government has basically declared "HTTPS/TLS Interception Considered Harmful". This is going to be interesting as all the major security load blanacer/appliances out there offer this as a standard service at this point.

userbinator 10y ago
Watch them change their opinion when the next leak of government information is determined to have happened over an unintercepted TLS connection...
saurik 10y ago
As far as I can tell this is targetted at what I would expect: "antivirus" and compliance tools built into firewalls, not reverse proxies built into load balancers.
- hueving 10y ago
  There is limited distinction there. Both terminate and re-establish TLS sessions.
  - saurik 10y ago
    I strongly disagree: who is in control, how it is implemented, and the purpose of the product are all both extremely important and fundamentally different between the two use cases. An "antivirus" filter ends up implemented as a man-in-the-middle attack between software I wrote and servers I contract and control, wherein protections that were actively put in place are stripped away in order to increase the attack surface. A reverse proxy is something that I actively implemented as part of my infrastructure, and I am contracting for it and paying for it like anything else I rent. It became "extra cool" to hate on reverse proxies recently while everyone hates CloudFlare (and to be clear: I've hated CloudFlare since before it was cool to hate on CloudFlare, and you should check my comment and submission history to verify if you don't believe ;P), but what people are essentially complaining about are "it encourages you to do something insecure", which I think is actually largely true only of CloudFlare, and you have to remember that the vast majority of developers who are using CloudFlare are doing so because they don't know much about network administration. But like, think about it this way: Apache is also an SSL termination point. The code in your website is likely 99% "transform a URL into SQL, then transform the result into HTML": it is just a dumb encoder/decoder, and pretty exactly analogous to an SSL termination point. If you really believe in true end-to-end encryption, then having this webserver in the middle is a bad idea: you should route requests directly to database servers and ensure that at no point does any non-database computer have a decrypted copy of the information that will eventually be rendered to the user. Another thing to think about: if you are heavily concerned that the CDN or reverse proxy service you are using has access to your traffic, realize that they are no different than using a hosted server. We don't even need to invoke "the cloud" for this one: unless I built the machine myself, I have no way of knowing that the server I am renting to run my database server on isn't running under a hypervisor that is allowing dumps of my decrypted memory. A reverse proxy is just another piece of rented infrastructure that someone else is in control of, and I don't see many people (though if you are, that's great, and then I'd be really happy ;P) insisting that everyone build their own computers and host them at their office, to make sure decrypted data never touches a computer owned and operated by someone else. The risk profile is also just so fundamentally different, in no small part due to these differences of "who is in control and how is it implemented". Let's say that CDNetworks is failing to check SSL certificates correctly: how exactly are you going to attack that against my website? I mean, sure: if you are a state-level actor, that is going to be easy, but otherwise, that's going to be extremely rough. An "antivirus" filter that, as described in this alert, is failing to correctly check SSL certificates... well, that opens you up to attack by amateur attackers that are "close to you", such as a stalker or an ex-spouse or a corporate competitor. You can't just claim "well, if I abstract away all of the details, they look the same"... you have to look at why you care and verify that you didn't strip away anything required and relevant.
    - nickik 10y ago
      > The code in your website is likely 99% "transform a URL into SQL, then transform the result into HTML" My live seems more trivil now.
    - JshWright 10y ago
      > If you really believe in true end-to-end encryption, then having this webserver in the middle is a bad idea: you should route requests directly to database servers and ensure that at no point does any non-database computer have a decrypted copy of the information that will eventually be rendered to the user. I can't say I haven't considered abusing PL/pgSQL to that point... ;)
      - _jal 10y ago
        I've actually seen that done - HTML straight out of a stored procedure. Really awesome <TR BGCOLOR=...><TD> stuff. I've mostly recovered from the experience, thanks, although I do have occasional flashbacks.
    - lmm 10y ago
      >An "antivirus" filter ends up implemented as a man-in-the-middle attack between software I wrote and servers I contract and control, wherein protections that were actively put in place are stripped away in order to increase the attack surface. A reverse proxy is something that I actively implemented as part of my infrastructure, and I am contracting for it and paying for it like anything else I rent. Those can equally happen the other way around. You might put an antivirus filter in place on your own network. Your hosting provider might set up their own reverse proxy. > But like, think about it this way: Apache is also an SSL termination point. The code in your website is likely 99% "transform a URL into SQL, then transform the result into HTML": it is just a dumb encoder/decoder, and pretty exactly analogous to an SSL termination point. If you really believe in true end-to-end encryption, then having this webserver in the middle is a bad idea: you should route requests directly to database servers and ensure that at no point does any non-database computer have a decrypted copy of the information that will eventually be rendered to the user. I agree with that - best to have the database and the web server on the same server, communicating via local socket or shared memory. Better still to compile the web layer and the data layer into the same unikernel. Current best practice (e.g. PCI-DSS) seems to accept having an (encrypted) network connection between distinct web and database servers, but only within a single physically secured rack. You're right that if you do the same thing with a load balancer (have it in the same physically secured rack as your web servers, encrypt connections between it and them) then it's no less secure, but I don't think people would want to be using a load balancer within a single rack anyway, since most of the point of a load balancer is availability and you want separate racks for that. Certainly a third-party reverse-proxy is a substantial weakening of security. > I don't see many people (though if you are, that's great, and then I'd be really happy ;P) insisting that everyone build their own computers and host them at their office, to make sure decrypted data never touches a computer owned and operated by someone else. I wouldn't go that far - AIUI PCI-DSS etc. require you have physically secured/access-controlled servers but allow you to rent space/power/cooling from a datacenter operator (though they don't get access to your rack, and possibly have to also be audited themselves). > Let's say that CDNetworks is failing to check SSL certificates correctly: how exactly are you going to attack that against my website? I mean, sure: if you are a state-level actor, that is going to be easy, but otherwise, that's going to be extremely rough. An "antivirus" filter that, as described in this alert, is failing to correctly check SSL certificates... well, that opens you up to attack by amateur attackers that are "close to you", such as a stalker or an ex-spouse or a corporate competitor. It's a different risk profile to be sure. Are you more worried about someone making a large-scale attack against the remote service you were connecting to, or just against the Internet in general? Or someone making a small-scale local attack? I think both can be serious; it's not just state-level actors but also large criminal rings or even political groups that would potentially have the resources to execute an attack against a particular service (e.g. a medical provider doing something politically contentious, or just anything that accepts credit cards / personal information).
  - deleted 10y ago
    [deleted]
  - drdaeman 10y ago
    When running on the same machine, reverse TLS-terminating proxy isn't different from built-in TLS implementation, except for done in "microservice-like" way. Most certainly, there's no difference whatsoever (or almost so, if the reverse proxy is "smart" and has some application logic in it) if your webapp doesn't speak HTTP natively, but talks FastCGI/uWSGI/SCGI protocols instead. Which is quite common. I mean, the actual interface (passing buffer pointers in the local process or passing same data via local UNIX sockets) shouldn't matter much, right? As long as interfaces have similar security properties and all the necessary data is passed around properly. A sufficiently privileged user can introspect both (with ltrace or tcpdump), and insufficiently privileged one can't do anything anyway.
  - jupp0r 10y ago
    Not necessarily. A common pattern for load balancers is terminating TLS and communicating with backends unencrypted/unauthenticated.
deleted 10y ago
[deleted]
micaksica 10y ago
I don't think that's the correct interpretation of this document. A better title would be "Most TLS Proxies Considered Harmful", not TLS interception as a whole. US-CERT is explicitly calling out that many appliances and applications often are weakening security vs. the common TLS clients, by way of not properly validating certificate chains by proxy for the client and otherwise masking SSL/TLS issues from being surfaced to the end user. I believe this is CERT trying to get software companies to get their act together on solving security-detrimental UX/UI issues vs. a blanket statement not to add TLS interception proxies to a corporate network.
- hannob 10y ago
  > I don't think that's the correct interpretation of this document. A better title would be "Most TLS Proxies Considered Harmful", not TLS interception as a whole. This is the usual reaction to this I see everywhere. It doesn't match with reality. Read the complete paper by Halderman et al [1]. It says pretty clearly that all existing products lower the security in one way or another: "Our grading scale focuses on the security of the TLS handshake and does not account for the additional HTTPS validation checks present in many browsers, such as HSTS, HPKP, OneCRL/CRLSets, certificate transparency validation, and OCSP must-staple. None of the products we tested supported these features." TLS Interception that doesn't degrade security is a myth that exists only in some parallel world fantasy. It's also not very surprising. Browser vendors have extremely qualified security teams that are involved in many ways in the development of stronger TLS protections. Nothing like that can be said about any of the vendors of interception products. [1] https://jhalderm.com/pub/papers/interception-ndss17.pdf https://jhalderm.com/pub/papers/interception-ndss17.pdf
  - mathw 10y ago
    Is better TLS interception actually possible? Is it just that it's usually badly implemented, or is it a fundamental weakening? Other than the unavoidable weakening of having another system seeing the unencrypted data, which is inherent in using one of these systems and can never go away.
    - 10y ago
Spooky23 10y ago
No, they haven't. They say pretty plainly that you need to ensure that such products provide correct certificate validation, since the client cannot do so reliably.
- Ajedi32 10y ago
  Well, it does also say: > In general, organizations considering the use of HTTPS inspection should carefully consider the pros and cons of such products before implementing So I think there's at least some element of "don't do this unless you have to" in there. But for the most part yeah, the alert seems to be more about ensuring the interception is done securely than about not doing it at all.
- jamiesonbecker 10y ago
  >> The US government has basically declared "HTTPS/TLS Interception Considered Harmful". > No, they haven't. The original report headline from US-CERT: "HTTPS Interception Weakens TLS Security" That seems pretty unambiguous, regardless of your personal feelings regarding CERT or whether it speaks for the broader US government.
  - Spooky23 10y ago
    Don't analyze a book by its cover!
deleted 10y ago
[deleted]
tptacek 10y ago
US-CERT has very little influence, even in regulated industries. As usual, CERT is saying something formally that has been axiomatic among practitioners for many years. The only conclusion I'd draw from this is about CERT. Obviously: what they're saying is broadly true. I also agree with the other commenters here who are pointing out that this is largely a response to endpoint TLS interception, and that the point is to focus attention on tools that intercept but don't validate certificates --- CERT is not telling every Fortune 500 company and every federal agency to stop deploying BlueCoat boxes. But the more important thing to push back on in your comment is the notion that a proclamation from CERT has any real meaning. It doesn't, even within FedGov IT.

userbinator 10y ago

Watch them change their opinion when the next leak of government information is determined to have happened over an unintercepted TLS connection...

saurik 10y ago

As far as I can tell this is targetted at what I would expect: "antivirus" and compliance tools built into firewalls, not reverse proxies built into load balancers.

hueving 10y ago
There is limited distinction there. Both terminate and re-establish TLS sessions.
- saurik 10y ago
  I strongly disagree: who is in control, how it is implemented, and the purpose of the product are all both extremely important and fundamentally different between the two use cases. An "antivirus" filter ends up implemented as a man-in-the-middle attack between software I wrote and servers I contract and control, wherein protections that were actively put in place are stripped away in order to increase the attack surface. A reverse proxy is something that I actively implemented as part of my infrastructure, and I am contracting for it and paying for it like anything else I rent. It became "extra cool" to hate on reverse proxies recently while everyone hates CloudFlare (and to be clear: I've hated CloudFlare since before it was cool to hate on CloudFlare, and you should check my comment and submission history to verify if you don't believe ;P), but what people are essentially complaining about are "it encourages you to do something insecure", which I think is actually largely true only of CloudFlare, and you have to remember that the vast majority of developers who are using CloudFlare are doing so because they don't know much about network administration. But like, think about it this way: Apache is also an SSL termination point. The code in your website is likely 99% "transform a URL into SQL, then transform the result into HTML": it is just a dumb encoder/decoder, and pretty exactly analogous to an SSL termination point. If you really believe in true end-to-end encryption, then having this webserver in the middle is a bad idea: you should route requests directly to database servers and ensure that at no point does any non-database computer have a decrypted copy of the information that will eventually be rendered to the user. Another thing to think about: if you are heavily concerned that the CDN or reverse proxy service you are using has access to your traffic, realize that they are no different than using a hosted server. We don't even need to invoke "the cloud" for this one: unless I built the machine myself, I have no way of knowing that the server I am renting to run my database server on isn't running under a hypervisor that is allowing dumps of my decrypted memory. A reverse proxy is just another piece of rented infrastructure that someone else is in control of, and I don't see many people (though if you are, that's great, and then I'd be really happy ;P) insisting that everyone build their own computers and host them at their office, to make sure decrypted data never touches a computer owned and operated by someone else. The risk profile is also just so fundamentally different, in no small part due to these differences of "who is in control and how is it implemented". Let's say that CDNetworks is failing to check SSL certificates correctly: how exactly are you going to attack that against my website? I mean, sure: if you are a state-level actor, that is going to be easy, but otherwise, that's going to be extremely rough. An "antivirus" filter that, as described in this alert, is failing to correctly check SSL certificates... well, that opens you up to attack by amateur attackers that are "close to you", such as a stalker or an ex-spouse or a corporate competitor. You can't just claim "well, if I abstract away all of the details, they look the same"... you have to look at why you care and verify that you didn't strip away anything required and relevant.
  - nickik 10y ago
    > The code in your website is likely 99% "transform a URL into SQL, then transform the result into HTML" My live seems more trivil now.
  - JshWright 10y ago
    > If you really believe in true end-to-end encryption, then having this webserver in the middle is a bad idea: you should route requests directly to database servers and ensure that at no point does any non-database computer have a decrypted copy of the information that will eventually be rendered to the user. I can't say I haven't considered abusing PL/pgSQL to that point... ;)
    - _jal 10y ago
      I've actually seen that done - HTML straight out of a stored procedure. Really awesome <TR BGCOLOR=...><TD> stuff. I've mostly recovered from the experience, thanks, although I do have occasional flashbacks.
  - lmm 10y ago
    >An "antivirus" filter ends up implemented as a man-in-the-middle attack between software I wrote and servers I contract and control, wherein protections that were actively put in place are stripped away in order to increase the attack surface. A reverse proxy is something that I actively implemented as part of my infrastructure, and I am contracting for it and paying for it like anything else I rent. Those can equally happen the other way around. You might put an antivirus filter in place on your own network. Your hosting provider might set up their own reverse proxy. > But like, think about it this way: Apache is also an SSL termination point. The code in your website is likely 99% "transform a URL into SQL, then transform the result into HTML": it is just a dumb encoder/decoder, and pretty exactly analogous to an SSL termination point. If you really believe in true end-to-end encryption, then having this webserver in the middle is a bad idea: you should route requests directly to database servers and ensure that at no point does any non-database computer have a decrypted copy of the information that will eventually be rendered to the user. I agree with that - best to have the database and the web server on the same server, communicating via local socket or shared memory. Better still to compile the web layer and the data layer into the same unikernel. Current best practice (e.g. PCI-DSS) seems to accept having an (encrypted) network connection between distinct web and database servers, but only within a single physically secured rack. You're right that if you do the same thing with a load balancer (have it in the same physically secured rack as your web servers, encrypt connections between it and them) then it's no less secure, but I don't think people would want to be using a load balancer within a single rack anyway, since most of the point of a load balancer is availability and you want separate racks for that. Certainly a third-party reverse-proxy is a substantial weakening of security. > I don't see many people (though if you are, that's great, and then I'd be really happy ;P) insisting that everyone build their own computers and host them at their office, to make sure decrypted data never touches a computer owned and operated by someone else. I wouldn't go that far - AIUI PCI-DSS etc. require you have physically secured/access-controlled servers but allow you to rent space/power/cooling from a datacenter operator (though they don't get access to your rack, and possibly have to also be audited themselves). > Let's say that CDNetworks is failing to check SSL certificates correctly: how exactly are you going to attack that against my website? I mean, sure: if you are a state-level actor, that is going to be easy, but otherwise, that's going to be extremely rough. An "antivirus" filter that, as described in this alert, is failing to correctly check SSL certificates... well, that opens you up to attack by amateur attackers that are "close to you", such as a stalker or an ex-spouse or a corporate competitor. It's a different risk profile to be sure. Are you more worried about someone making a large-scale attack against the remote service you were connecting to, or just against the Internet in general? Or someone making a small-scale local attack? I think both can be serious; it's not just state-level actors but also large criminal rings or even political groups that would potentially have the resources to execute an attack against a particular service (e.g. a medical provider doing something politically contentious, or just anything that accepts credit cards / personal information).
- deleted 10y ago
  [deleted]
- drdaeman 10y ago
  When running on the same machine, reverse TLS-terminating proxy isn't different from built-in TLS implementation, except for done in "microservice-like" way. Most certainly, there's no difference whatsoever (or almost so, if the reverse proxy is "smart" and has some application logic in it) if your webapp doesn't speak HTTP natively, but talks FastCGI/uWSGI/SCGI protocols instead. Which is quite common. I mean, the actual interface (passing buffer pointers in the local process or passing same data via local UNIX sockets) shouldn't matter much, right? As long as interfaces have similar security properties and all the necessary data is passed around properly. A sufficiently privileged user can introspect both (with ltrace or tcpdump), and insufficiently privileged one can't do anything anyway.
- jupp0r 10y ago
  Not necessarily. A common pattern for load balancers is terminating TLS and communicating with backends unencrypted/unauthenticated.

deleted 10y ago

[deleted]

micaksica 10y ago

I don't think that's the correct interpretation of this document. A better title would be "Most TLS Proxies Considered Harmful", not TLS interception as a whole. US-CERT is explicitly calling out that many appliances and applications often are weakening security vs. the common TLS clients, by way of not properly validating certificate chains by proxy for the client and otherwise masking SSL/TLS issues from being surfaced to the end user. I believe this is CERT trying to get software companies to get their act together on solving security-detrimental UX/UI issues vs. a blanket statement not to add TLS interception proxies to a corporate network.

hannob 10y ago
> I don't think that's the correct interpretation of this document. A better title would be "Most TLS Proxies Considered Harmful", not TLS interception as a whole. This is the usual reaction to this I see everywhere. It doesn't match with reality. Read the complete paper by Halderman et al [1]. It says pretty clearly that all existing products lower the security in one way or another: "Our grading scale focuses on the security of the TLS handshake and does not account for the additional HTTPS validation checks present in many browsers, such as HSTS, HPKP, OneCRL/CRLSets, certificate transparency validation, and OCSP must-staple. None of the products we tested supported these features." TLS Interception that doesn't degrade security is a myth that exists only in some parallel world fantasy. It's also not very surprising. Browser vendors have extremely qualified security teams that are involved in many ways in the development of stronger TLS protections. Nothing like that can be said about any of the vendors of interception products. [1] https://jhalderm.com/pub/papers/interception-ndss17.pdf https://jhalderm.com/pub/papers/interception-ndss17.pdf
- mathw 10y ago
  Is better TLS interception actually possible? Is it just that it's usually badly implemented, or is it a fundamental weakening? Other than the unavoidable weakening of having another system seeing the unencrypted data, which is inherent in using one of these systems and can never go away.
  - 10y ago

Spooky23 10y ago

No, they haven't. They say pretty plainly that you need to ensure that such products provide correct certificate validation, since the client cannot do so reliably.

Ajedi32 10y ago
Well, it does also say: > In general, organizations considering the use of HTTPS inspection should carefully consider the pros and cons of such products before implementing So I think there's at least some element of "don't do this unless you have to" in there. But for the most part yeah, the alert seems to be more about ensuring the interception is done securely than about not doing it at all.
jamiesonbecker 10y ago
>> The US government has basically declared "HTTPS/TLS Interception Considered Harmful". > No, they haven't. The original report headline from US-CERT: "HTTPS Interception Weakens TLS Security" That seems pretty unambiguous, regardless of your personal feelings regarding CERT or whether it speaks for the broader US government.
- Spooky23 10y ago
  Don't analyze a book by its cover!

deleted 10y ago

[deleted]

tptacek 10y ago

US-CERT has very little influence, even in regulated industries. As usual, CERT is saying something formally that has been axiomatic among practitioners for many years. The only conclusion I'd draw from this is about CERT. Obviously: what they're saying is broadly true. I also agree with the other commenters here who are pointing out that this is largely a response to endpoint TLS interception, and that the point is to focus attention on tools that intercept but don't validate certificates --- CERT is not telling every Fortune 500 company and every federal agency to stop deploying BlueCoat boxes. But the more important thing to push back on in your comment is the notion that a proclamation from CERT has any real meaning. It doesn't, even within FedGov IT.