9 ms·
TLS Termination for Network Load Balancers
- zob_cloud 8y agoElastic Load Balancing blog post about TLS Termination on Network Load Balancers! This includes the ability to preserve source IP to your targets while terminating / offloading TLS. Checkout the demo https://network.exampleloadbalancer.com/nlbtls_demo.html https://network.exampleloadbalancer.com/nlbtls_demo.html
- jamescun 8y ago> This will free your backend servers from the compute-intensive work of encrypting and decrypting all of your traffic https://istlsfastyet.com/ https://istlsfastyet.com/
- lenova 8y agoDumb question, but has the industry decided on what they want to call TLS/HTTPS? Majority of people I talk to still refer to them as SSL certs...
- atombender 8y agoOld habits die hard. The current standard is TLS, which superceded SSL back in 1999. Uptake and awareness was initially slow, and SSL itself wasn't deprecated until 2015. It doesn't help that a bunch of projects (OpenSSL being a prominent one) still have "SSL" in their name. Technically, they're not "SSL certs". They're X.509 certs. Certificates show up in many places unrelated to TLS/SSL.
- tialaramex 8y agoWell, the things people want aren't (generally) just X.509 certificates. If you expect your certificates to work in common client software like web browsers or some random Python client code a third party is writing then you're going to want: * PKIX, currently RFC 5280 plus revisions, the Internet's chosen standard for how X.509 should be implemented. PKIX says a bunch of things about what you should or should not write in the X.509 certificate, you will probably have better luck conforming as much as possible even if you don't care about: * The Web PKI. A Public Key Infrastructure has Certificate Authorities (plenty of those across all of X.509 or you can roll your own) but they're trusted by Relying Parties (parties who are _relying_ on the certificate's attestation to be true) and you probably want certificates that will be trusted by most RPs. Grandma doesn't know what the X.500 directory system is, or who a Certificate Authority is, but she does use Safari, and Safari trusts certain CAs on her behalf as does macOS. So you're going to want certs from one of those CAs. The Web PKI is a loose name for (despite that word "Web") the PKI covering SSL/TLS services on the Internet. * You may want to get more specific. Although the Baseline Requirements which say roughly how a Certificate Authority should work are agreed across industry, each of the Major Trust Stores has their own additional rules. You probably care about all of them. They roughly correspond to operating system vendors. Microsoft and Apple (for their browsers and operating systems), then Mozilla (for Firefox on all platforms, and for Free Unix systems), Google (mostly just for Android, not Chrome), and then runners up Oracle (for Java), then a long tail of people including Nintendo and all those crappy in-car entertainment systems people... For example, probably fifty people in the whole world have ever bothered trying to use the Web from a Nintendo WiiU (not to be confused with the popular Wii) console. If they try now though lots of things don't work. Because Let's Encrypt isn't trusted by the WiiU, and since it's an end-of-life product, probably never will be. But "SSL Certs" gets across what you mean pretty well, only pedants are going to insist it's wrong, and unless you're currently playing "Um, Actually" they should cut it out.
- Edmond 8y agoit is TLS...SSL is just much better known and likely would be around until written references to SSL go away..it would be a while :) SSL references "sockets", which is still where TLS is implemented but TLS is conceptually about transport security independent of implementation (ala sockets)....for instance consider this TLS inspired solution (https://www.cipheredtrust.com/doc/ https://www.cipheredtrust.com/doc/), it has nothing to do with sockets.
- naniwaduni 8y agoFor the most part, it's called SSL. Hell, it's even more taxonomically coherent, take it from the bear: > So basically, the “SSL vs TLS” taxonomy represents the ownership change in 1999, but not the technical details of the protocols. From a purely technical point of view, lumping together SSL-3.0 with SSL-2.0 but separating it from the TLS versions is about as wrong as, in biology, classifying whales as a kind of fish (despite what Herman Melville wrote about them). > Therefore, I tend to use “SSL” (or “SSL/TLS”) to designate the whole conceptual family of protocols, from SSL-1.0 to TLS-1.2. This is the main reason why BearSSL is called BearSSL and not BearTLS. > A secondary reason is more about marketing. People who know what TLS is also know what SSL is, but not the other way round. For instance, Web site owners look for a “SSL certificate”, not a “TLS certificate”. By using the “SSL” acronym, I thus reach a wider audience, even if it is at the cost of irritating some of the more pickish taxonomists (and I am totally ready to out-pedant them if need be, as demonstrated above). (https://bearssl.org/naming.html https://bearssl.org/naming.html)
- kevin_thibedeau 8y agoPolarSSL took the plunge and renamed. It isn't impossible.
- naniwaduni 8y agoTLS is our octopi.
- isodude 8y agoMy customers won't stop calling them SLL Certificates. Drives me mad.
- aphextron 8y ago> Dumb question, but has the industry decided on what they want to call TLS/HTTPS? Majority of people I talk to still refer to them as SSL certs... They are all independent terms. Secure Socket Layers (SSL) is an outdated term that refers to any network data transfer over Transmission Control Protocol (TCP) using the Transport Layer Security (TLS) protocol, of which the Hypertext Transfer Protocol Secure (HTTPS) protocol is an application layer implementation.
- azinman2 8y agoSSL added and removed here :)
- atombender 8y agoSo AWS didn't have TLS termination until now? I'm still waiting for GCP to support more than ten (!) certs per load balancer, which seems like a ridiculous limit when you consider the need to serve dozens or hundreds of low-traffic customer domains on a single domain. If you're on Kubernetes, they're basically asking you to split your ingresses up just to avoid the limit. We ended up going straight to running our own in-cluster load balancer (Traefik, which is meh, but works okay) with Let's Encrypt so we don't need to provision anything at all. It's so much nicer than fiddling with manual cert registration. I really wish cloud providers such as Google would get on board with Let's Encrypt already.
- akerl_ 8y agoThey supported it on ELBs and ALBs, but not NLBs.
- helper 8y agoThey have supported TLS termination for https connections for a long time. This adds support for TLS on TCP connections. The NLB load balancer also allows you to keep the same set of pubic IP addresses for your load balancer endpoints.
- deleted 8y ago[deleted]
- xyzzy123 8y agoI’m not sure about gcp but on aws you can have domain validated acm certs which are arguably more convenient than LE certs. aws will transparently renew them for you if they are associated with a load balancer or cloudfront. no moving parts required in your infra.
- briffle 8y agoWe use cert-manager to get letsencrypt certs in GKE, and automatically push them to the Google global load balacer's. https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager
- 8y ago
- DGAP 8y agoThis further reinforces the pattern of terminating TLS at the LB. While this is generally justifiable, it does decrease defense-in-depth.
- jontro 8y agoAfter choosing the certificate and the policy, I click Next:Configure Routing. I can choose the communication protocol (TCP or TLS) that will be used between my NLB and my targets. If I choose TLS, communication is encrypted; this allows you to make use of complete end-to-end encryption in transit: You can still communicate over ssl internally. What do you mean with it will decreases the defence?
- scarface74 8y agoBut then what’s the purpose of terminating at the LB? The two advantages are that it takes the load off of your web server having to decrypt traffic and autorenwal of SSL certs. Once you have to manage your own certs and decrypt at your server, both of those advantages are removed.
- jontro 8y agoHaving the public facing certificate managed by acm and not worrying about deploying those on the instances is a win. I'm not sure how the certificate validation is done LB <-> instance, but if you can have a private CA and use that for intra-communication it would still keep everything encrypted properly
- dc_gregory 8y agoAuto generated self signed certs on the servers
- scarface74 8y agoWhat’s the threat model for someone intercepting traffic between a load balancer and an EC2 instance?
- scarface74 8y agoUnfortunately, this still wouldn’t have solved my biggest pain point at the last company I worked for. You can’t use TLS termination at the load balancer when you require HIPAA Compliance. If they had some kind of way to use ACM generated SSL certs on the VM and had a method for autorenewal that would be ideal.
- nkrumm 8y agoEncryption is an "addressable implementation" in the final Security Rule [1]. Practically this means you are not required to encrypt your data (e.g., between LB and VM) "if the entity decides that the addressable implementation specification is not reasonable and appropriate [...]". https://www.hhs.gov/hipaa/for-professionals/faq/2001/is-the-use-of-encryption-mandatory-in-the-security-rule/index.html https://www.hhs.gov/hipaa/for-professionals/faq/2001/is-the-...
- jontro 8y agoAnd also it looks like it supports encrypted traffic between the LB and the instance as shown in the guide.
- e1g 8y agoAWS LB does not validate backend certificates, so you can put a self-signed cert on the instance. Heck, even if the cert expires it will still work, and make LB<->EC2 connection technically encrypted. Yay compliance.
- xyzzy123 8y agoYou can actually turn this on (for classic ELBs), but it locks to a specific cert (rather than a CA). So yeah no one cares about expiry but the backend does have to present that cert. The thing to look for is "Enable backend authentication". I would question whether this is a problem though; basically if someone is in a position to MITM traffic in your AWS VPC this would indicate a compromise of AWS at a fundamental level (or loss of your AWS control plane).
- clon 8y agoI was pretty excited until I saw the screenshot where you would have have to choose the certificate. If you run a multi-tenant system that services N "vanity" domains (where N is around the same number as your users on a $5/month plan), there is still no service in AWS to do transparent TLS termination for a reasonable cost. Which is a pity, since it really costs almost nothing to generate these certificates.
- helper 8y agoWe use wildcard certs (*.example.com) so each customer can have vanity-customer-name.example.com domains. I think our model is fairly common for multi-tenant domain name segregated systems.
- clon 8y agoWe do that as well for the cheaper packages. But serious users insist on a our-service.client.com "vanity" scheme, which is easy enough with CNAME records, apart from TLS that adds significant cost. For example, Application Load Balancers (ALB) come with a limit of max 25 certificates, which is non-starter for us. So you cannot avoid terminating your TLS in nginx/caddy etc (it also needs to be HA of course) and then hit another LB that stands in front of the actual service. You end up with a 2-layer LB architecture that adds cost and complexity.
- web007 8y agoCan you bundle your certificates the way Cloudflare does? IIRC if you connect to a CF endpoint you will get a certificate with N arbitrary hostnames that are being served by that endpoint. I don't know what the SNI hostname limit is, but it's probably stupidly high. Multiply that by 25 and it may be tenable?
- elithrar 8y agoCloudflare has a service that does this automatically - they call it "SSL for SaaS" - and you can provision custom/vanity hostnames onto their own certificates within a minute or so + push to the edge. You should reach out to https://twitter.com/prdonahue https://twitter.com/prdonahue (PM), who can share the details. (Used to work there, now at GCP)
- luhn 8y agoThey don't mention it in the article, but TLS comes at an additional cost. Same price per hour and per LCU, but an LCU gets you less with TLS. Standard LCU: - 800 new non-TLS connections or flows per second. - 100,000 active non-TLS connections or flows (sampled per minute). - 1 GB per hour for EC2 instances, containers and IP addresses as targets. TLS LCU: - 50 new TLS connections or flows per second. - 3,000 active TLS connections or flows (sampled per minute). - 1 GB per hour for EC2 instances, containers and IP addresses as targets. https://aws.amazon.com/elasticloadbalancing/pricing/ https://aws.amazon.com/elasticloadbalancing/pricing/
- deleted 8y ago[deleted]
- NotAnEconomist 8y agoJust because I found it interesting: - 16x cost for new connections or flows per second. - 32x cost for active connections or flows. - 1x cost for traffic carried in GB per hour, which you'd expect.
- buzzdenver 8y agoThat's still double the connections compared to having an ALB which only gives you 25/s/LCU. Plus an NLB is 0.6 cents/hour/LCU in us-east-1 vs 0.8 for an ALB.
- illumin8 8y agoThere is a great mini-thread by a principal engineer at AWS describing why this is so amazing: https://twitter.com/colmmacc/status/1088510453767000064 https://twitter.com/colmmacc/status/1088510453767000064
- barbadosmercury 8y agoHow do they ensure the target machines see the original IP address and port?
- deleted 8y ago[deleted]
- ec109685 8y agoThey encapsulate the packet, including data about the original IP address, as it is sent to the hypervisor on the ec2 instance. Then the hypervisor reverses the process and creates a packet with the original IP address as a source when it is forwarded into the vm.
- weitzj 8y agoThat’s great news. Does anybody know if they support ALPN to announce h2 as the http2 protocol? They don’t do this for ELBs, I would hope for NLBs. If they do this, this would finally be the sane solution how to use NLBs + ACM to expose gRPC services at the edge. Other solution might maybe be (as popping up here in the threads): Use NLB + ACM and let it do MITM and then forward the traffic to a service with a self signed certificate. The hope here would be that the NLB would pick up the ALPN header from your backend service and communicate it transparently to the outside world. Don’t know if this architecture makes sense. Anyways, I just would love gRPC capable secure workloads on the NLB with ACM. Doing Self signed certificates or LetsEncrypt works already - the Gamechanger is to use ACM + ALPN and by that also having upstream http2 traffic behind the NLB (compared to the ALB, which only supports upstream http1)
- philsnow 8y ago> After choosing the certificate and the policy, I click Next:Configure Routing. I can choose the communication protocol (TCP or TLS) that will be used between my NLB and my targets. If I choose TLS, communication is encrypted; this allows you to make use of complete end-to-end encryption in transit Wait. That is not what "end to end" means. This is more like "piecewise end-to-end", and it is not the same thing as "end to end". Now, people are going to say that "it doesn't matter since the only party you're disclosing the comms to is AWS, who you already implicitly depend on because you're running all your stuff under their hypervision". That's true. (ofc now there are 2 places in AWS's infra where the comms are in plaintext, so now an adversary has two teams of engineers/opsen to social engineer / compromise, and will win if they break either one). I'm reacting to the dilution of the term "end to end" here, because it's a vital concept.