4 ms·

Do you mean to say that your biggest frustration with HTTPS on .internal is that it requires a private certificate authority? Because I'm running plain HTTP to

Do you mean to say that your biggest frustration with HTTPS on .internal is that it requires a private certificate authority? Because I'm running plain HTTP to .internal sites and it works fine.

lysace 2y ago
There's some every packet shall be encrypted, even in minimal private VPCs lore going on. I'm blaming PCI-DSS.
- bruce511 2y ago
  The big problem with running unencrypted HTTP on a LAN is that it's terribly easy for (most) LANs to be compromised. Let's start with the obvious; wifi. If you're visiting a company and ask the receptionist for the wifi password you'll likely get it. Next are eternity ports. Sitting waiting in a meeting room, plug your laptop into the ethernet port and you're in. And of course it's not just hardware, any software running on any machine makes the LAN just as vulnerable. Sure, you can design a LAN to be secure. You can make sure there's no way to get onto it. But the -developer- and -network maintainer- are 2 different guys, or more likely different departments. As a developer are you convinced the LAN will be as secure in 10 years as it is today? 5 years? 1 year after that new intern arrives and takes over maintainence 6 weeks in? What starts out as "minimal private VPC" grows, changes, is fluid. Treating it as secure today is one thing. Trusting it to remain secure 10 years from now is another. In 99.9% of cases your LAN traffic should be secure. This us the message -developers- need to hear. Don't rely on some other department to secure your system. Do it yourself.
  - gorgoiler 2y ago
    Well said. I used to be of the mindset that if I ran VLANs I could at least segregate the good guys from the evil AliExpress wifi connected toasters. Now everything feels like it could become hostile at any moment and so, on that basis, we all share the same network with shields up as if it were the plain, scary Internet. It feels a lot safer. I guess my toaster is going to hack my printer someday, but at least it won’t get into my properly-secured laptop that makes no assumptions the local network is “safe”.
  - slimsag 2y ago
    Also, make sure your TLS certificates are hard-coded/pinned in your application binary. Just like the network, you really cannot trust what is happening on the user's system. This way you can ensure you as the developer have full control over your applications' network communication; by requiring client certificates issued by a CA you control, you can assert there is no MITM even if a sysadmin, user, or malware tries to install a proxy root CA on the system. Finally, you can add binary obfuscation / anticheat mechanisms used commonly in video games to ensure that even if someone is familiar with the application in question they cannot alter the certificates your application will accept. Lots of e.g. mobile banking apps, etc. do this for maximal security guarantees.
    - PokestarFan 2y ago
      At some point you have to wonder if your app even matters that much.
      - bruce511 2y ago
        The App probably not. The server maybe, the data probably.
    - frogsRnice 2y ago
      Pinning is very complex, there is always the chance that you forget to update the pins and perform a denial of service against your own users. At the point where the device itself is compromised, you can’t really assert to anything. Furthermore, there is always the risk that your developers implement pinning incorrectly and introduce a chain validation failure. Lots of apps use the anticheat/obfuscation mechanisms added by mobile apps are also trivial to bypass using instrumentation - ie frida codeshare. I know you aren’t implying that people should use client side controls to protect an app running on a device and an environment that they control, but in my experience even some technical folk will try and to do this
    - swiftcoder 2y ago
      In practice pinning tends to be very "best effort", if not outright disadvantageous. All our apps had to auto-disable pinning less than a year after the build date, because if the user hadn't updated the app by the time we had to renew all our certs... they'd be locked out. Also dealt with the fallout from a lovely little internet-of-things device that baked cert pinning into the firmware, but after a year on store shelves the clock battery ran out, so they booted up in 1970 and decided the pinned certs wouldn't become valid for ~50 years :D
  - Spooky23 2y ago
    The big issue with encrypted HTTP on the local LAN is that you’re stuck running a certificate authority, ignoring TLS validation, or exposing parts of your network in the name of transparency. Running certificate authority is one of those a minute to learn, lifetime to master scenarios. You are often trading “people can sniff my network scenario” to a “compromise the CA someone setup 10 years ago that we don’t touch” scenario.
    - bruce511 2y ago
      I agree that setting up a self-signed CA is hard, and harder to keep going. However DNS challenge allow for you to map an internal address to an IP number. The only real information that leaks is the subnet address of my LAN. And given the choice of that or unencrypted traffic I'll take that all day long.
  - xp84 2y ago
    For most purposes, when wishing for non-HTTPS, we are talking about development or maybe a staging server of some sort. Maybe if we had state secrets people would be trying to plug into the lan to snoop the traffic, but for 99.99% of developers the traffic between a testing instance and them is the most worthless thing ever. Worst case you might find out what features we will release to the app in 2 weeks. The conflation of “SSL” with “cybersecurity” is unfortunate.
- kortilla 2y ago
  Hoping datacenter to datacenter links are secure is how the NSA popped Google. Turn on crypto, don’t be lazy
  - otabdeveloper4 2y ago
    Pretty sure state-level actors sniffing datacenter traffic is literally the very last of your security issues. This kind of theater actively harms your organization's security, not helps it. Do people not do risk analysis anymore?
    - shawnz 2y ago
      Taking defense in depth measures like using https on the local network is "theatre" that "actively harms your organization's security"? That seems like an extreme opinion to me. Picking some reasonable best practices like using https everywhere for the sake of maintaining a good security posture doesn't mean that you're "not doing risk analysis".
      - the8472 2y ago
        I have seen people disabling all cert validation in an application because SSL was simultaneously required and no proper CA was provided for internal things. The net effect was thus that even the traffic going to the internet was no longer validated.
    - soraminazuki 2y ago
      NSA sniffs all traffic through various internet choke points in what's known as upstream surveillance. It's not just data center traffic. https://www.eff.org/pages/upstream-prism https://www.eff.org/pages/upstream-prism These kind of risks are obvious, real, and extensively documented stuff. I can't imagine why anyone serious about improving security for everyone would want to downplay and ridicule it.
    - TimTheTinker 2y ago
      Found the NSA goon. Seriously, your statement is demonstrably wrong. That's exactly the sort of traffic the NSA actively seeks to exploit.
- yarg 2y ago
  Blame leaked documents from the intelligence services. No one really bothered until it was revealed that organisations like the NSA were exfiltrating unencrypted internal traffic from companies like Google with programs like PRISM.
  - baq 2y ago
    Echelon was known about before Google was even a thing. I remember people adding Usenet headers with certain keywords. Wasn’t much, but it was honest work.
- laz 2y ago
  Exactly what an NSA puppet account would say! Don't believe the hype. Remember the smiley from "SSL added and removed here" https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-here-nsa-smiley/ https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...
  - lysace 2y ago
    This "NSA puppet" is all for encrypting traffic between networks. ;-)
- unethical_ban 2y ago
  That's some "it's okay to keep my finger on the trigger when the gun is unloaded" energy.
j1elo 2y ago
Try running anything more complicated than a plain and basic web server! See what happens if you attempt to serve something that browsers deem to require a mandatory "Secure Context", so they will reject running it when using HTTP. For example, you won't be able to run internal videocalls (no access to webcams!), or a web page able to scan QR codes. Here's the full list: * https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts/features_restricted_to_secure_contexts https://developer.mozilla.org/en-US/docs/Web/Security/Secure... A true hassle for internal testing between hosts, to be honest. I just cannot run an in-development video app on my PC and connect from a phone or laptop to do some testing, without first worrying about certs at a point in development where they are superfluous and a loss of time.
- akira2501 2y ago
  localhost is a secure context. so.. presumably we're just waiting for .internal to be added to the white list.
  - JonathonW 2y ago
    Unlikely. Localhost can be a secure context because localhost traffic doesn't leave your local machine; .internal names have no guarantees about where they go (not inconceivable that some particularly "creative" admin might have .internal names that resolve to something on the public internet).
    - Wicher 2y ago
      One can resolve "localhost" (even via an upstream resolver) to an arbitrary IP address. At least on my Linux system "localhost" only seems to be specially treated by systemd-resolved (with a cursory attempt I didn't succeed in getting it to use an upstream resolver for it). So it's not a rock-hard guarantee that traffic to localhost never leaves your system. It would be unconventional and uncommon for it to, though, except for the likes of us who like to ssh-tunnel all kinds of things on our loopback interfaces :-) The sweet spot of security vs convenience, in the case of browsers and awarding "secure origin status" for .internal, could perhaps be on a dynamic case by case basis at connect time: - check if it's using a self-signed cert - offer TOFU procedure if so - if not, verify as usual Maaaaybe check whether the connection is to an RFC1918 private range address as well. Maybe. It would break proxying and tunneling. But perhaps that'd be a good thing. This would just be for browsers, for the single purpose of enabling things like serviceworkers and other "secure origin"-only features, on this new .internal domain.
this_user 2y ago
A lot of services default to HTTPS. For instance, try setting up an internal Gitlab instance with runners, pipelines, and package/container registries that actually works. It's an absolute nightmare, and some things outright won't work. And if you want to pull images from HTTP registries with Docker, you have enable that on every instance for each registry separately. You'd be better off registering a real domain, using Let's Encrypt with the DNS challenge, and setting up an internal DNS for your services. That is literally an order of magnitude less work than setting up HTTP.
IshKebab 2y ago
A lot of modern web features now require HTTPS.
jve 2y ago
I consider HTTPS to be easier to run - you get less trouble in the end. As mentioned, some browser features are HTTPS only. You get security warnings on HTTP. Many tools now default to HTTPS by default - like newer SQL Server drivers. Dev env must resemble prod very closely so having HTTP in DEV and HTTPS in prod is asking for pain and trouble. It forces you to have some kind of expiration registry/monitoring and renewal procedures. And you happen to go throught dev env first and gain confidence and then prod. Then there are systems where client certificate is mandatory and you want to familiarize yourself already in dev/test env. Some systems even need additional configuration to allow OAuth via HTTP and that makes me feel dirty thus I rather not do it. Why do it if PROD won't have HTTP? And if one didn't know such configuration must be done, you'd be troubleshooting that system and figuring out why it doesn't work with my simple setup? Yeah, we have internal CA set up, so issuing certs are pretty easy and mostly automated and once you go HTTPS all in, you get the experience why/how things work and why they may not and got more experience to troubleshoot HTTPS stuff. You have no choice actually - the world has moved to TLS secured protocols and there is no way around getting yourself familiar with security certificates.
- 8organicbits 2y ago
  At my first job out of college we built an API and a couple official clients for it. The testing endpoint used self-signed certs so we had to selectively configure clients to support it. Right before product launch we caught that one of our apps was ignoring certificate verification in production too due to a bug. Ever since then I've tried to run publicly valid certificates on all endpoints to eliminate those classes of bugs. I still run into accidentally disabled cert validation doing security audits, it's a common mistake.
8organicbits 2y ago
If you're on a laptop or phone that switches between WiFi networks then you are potentially spilling session cookies and other data unencrypted onto other networks that also happen to resolve .internal. HTTPS encrypts connections, but it also authenticates servers. The later is important too.

Do you mean to say that your biggest frustration with HTTPS on .internal is that it requires a private certificate authority? Because I'm running plain HTTP to .internal sites and it works fine.

lysace 2y ago

There's some every packet shall be encrypted, even in minimal private VPCs lore going on. I'm blaming PCI-DSS.

bruce511 2y ago
The big problem with running unencrypted HTTP on a LAN is that it's terribly easy for (most) LANs to be compromised. Let's start with the obvious; wifi. If you're visiting a company and ask the receptionist for the wifi password you'll likely get it. Next are eternity ports. Sitting waiting in a meeting room, plug your laptop into the ethernet port and you're in. And of course it's not just hardware, any software running on any machine makes the LAN just as vulnerable. Sure, you can design a LAN to be secure. You can make sure there's no way to get onto it. But the -developer- and -network maintainer- are 2 different guys, or more likely different departments. As a developer are you convinced the LAN will be as secure in 10 years as it is today? 5 years? 1 year after that new intern arrives and takes over maintainence 6 weeks in? What starts out as "minimal private VPC" grows, changes, is fluid. Treating it as secure today is one thing. Trusting it to remain secure 10 years from now is another. In 99.9% of cases your LAN traffic should be secure. This us the message -developers- need to hear. Don't rely on some other department to secure your system. Do it yourself.
- gorgoiler 2y ago
  Well said. I used to be of the mindset that if I ran VLANs I could at least segregate the good guys from the evil AliExpress wifi connected toasters. Now everything feels like it could become hostile at any moment and so, on that basis, we all share the same network with shields up as if it were the plain, scary Internet. It feels a lot safer. I guess my toaster is going to hack my printer someday, but at least it won’t get into my properly-secured laptop that makes no assumptions the local network is “safe”.
- slimsag 2y ago
  Also, make sure your TLS certificates are hard-coded/pinned in your application binary. Just like the network, you really cannot trust what is happening on the user's system. This way you can ensure you as the developer have full control over your applications' network communication; by requiring client certificates issued by a CA you control, you can assert there is no MITM even if a sysadmin, user, or malware tries to install a proxy root CA on the system. Finally, you can add binary obfuscation / anticheat mechanisms used commonly in video games to ensure that even if someone is familiar with the application in question they cannot alter the certificates your application will accept. Lots of e.g. mobile banking apps, etc. do this for maximal security guarantees.
  - PokestarFan 2y ago
    At some point you have to wonder if your app even matters that much.
    - bruce511 2y ago
      The App probably not. The server maybe, the data probably.
  - frogsRnice 2y ago
    Pinning is very complex, there is always the chance that you forget to update the pins and perform a denial of service against your own users. At the point where the device itself is compromised, you can’t really assert to anything. Furthermore, there is always the risk that your developers implement pinning incorrectly and introduce a chain validation failure. Lots of apps use the anticheat/obfuscation mechanisms added by mobile apps are also trivial to bypass using instrumentation - ie frida codeshare. I know you aren’t implying that people should use client side controls to protect an app running on a device and an environment that they control, but in my experience even some technical folk will try and to do this
  - swiftcoder 2y ago
    In practice pinning tends to be very "best effort", if not outright disadvantageous. All our apps had to auto-disable pinning less than a year after the build date, because if the user hadn't updated the app by the time we had to renew all our certs... they'd be locked out. Also dealt with the fallout from a lovely little internet-of-things device that baked cert pinning into the firmware, but after a year on store shelves the clock battery ran out, so they booted up in 1970 and decided the pinned certs wouldn't become valid for ~50 years :D
- Spooky23 2y ago
  The big issue with encrypted HTTP on the local LAN is that you’re stuck running a certificate authority, ignoring TLS validation, or exposing parts of your network in the name of transparency. Running certificate authority is one of those a minute to learn, lifetime to master scenarios. You are often trading “people can sniff my network scenario” to a “compromise the CA someone setup 10 years ago that we don’t touch” scenario.
  - bruce511 2y ago
    I agree that setting up a self-signed CA is hard, and harder to keep going. However DNS challenge allow for you to map an internal address to an IP number. The only real information that leaks is the subnet address of my LAN. And given the choice of that or unencrypted traffic I'll take that all day long.
- xp84 2y ago
  For most purposes, when wishing for non-HTTPS, we are talking about development or maybe a staging server of some sort. Maybe if we had state secrets people would be trying to plug into the lan to snoop the traffic, but for 99.99% of developers the traffic between a testing instance and them is the most worthless thing ever. Worst case you might find out what features we will release to the app in 2 weeks. The conflation of “SSL” with “cybersecurity” is unfortunate.
kortilla 2y ago
Hoping datacenter to datacenter links are secure is how the NSA popped Google. Turn on crypto, don’t be lazy
- otabdeveloper4 2y ago
  Pretty sure state-level actors sniffing datacenter traffic is literally the very last of your security issues. This kind of theater actively harms your organization's security, not helps it. Do people not do risk analysis anymore?
  - shawnz 2y ago
    Taking defense in depth measures like using https on the local network is "theatre" that "actively harms your organization's security"? That seems like an extreme opinion to me. Picking some reasonable best practices like using https everywhere for the sake of maintaining a good security posture doesn't mean that you're "not doing risk analysis".
    - the8472 2y ago
      I have seen people disabling all cert validation in an application because SSL was simultaneously required and no proper CA was provided for internal things. The net effect was thus that even the traffic going to the internet was no longer validated.
  - soraminazuki 2y ago
    NSA sniffs all traffic through various internet choke points in what's known as upstream surveillance. It's not just data center traffic. https://www.eff.org/pages/upstream-prism https://www.eff.org/pages/upstream-prism These kind of risks are obvious, real, and extensively documented stuff. I can't imagine why anyone serious about improving security for everyone would want to downplay and ridicule it.
  - TimTheTinker 2y ago
    Found the NSA goon. Seriously, your statement is demonstrably wrong. That's exactly the sort of traffic the NSA actively seeks to exploit.
yarg 2y ago
Blame leaked documents from the intelligence services. No one really bothered until it was revealed that organisations like the NSA were exfiltrating unencrypted internal traffic from companies like Google with programs like PRISM.
- baq 2y ago
  Echelon was known about before Google was even a thing. I remember people adding Usenet headers with certain keywords. Wasn’t much, but it was honest work.
laz 2y ago
Exactly what an NSA puppet account would say! Don't believe the hype. Remember the smiley from "SSL added and removed here" https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-here-nsa-smiley/ https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...
- lysace 2y ago
  This "NSA puppet" is all for encrypting traffic between networks. ;-)
unethical_ban 2y ago
That's some "it's okay to keep my finger on the trigger when the gun is unloaded" energy.

j1elo 2y ago

Try running anything more complicated than a plain and basic web server! See what happens if you attempt to serve something that browsers deem to require a mandatory "Secure Context", so they will reject running it when using HTTP. For example, you won't be able to run internal videocalls (no access to webcams!), or a web page able to scan QR codes. Here's the full list: * https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts/features_restricted_to_secure_contexts https://developer.mozilla.org/en-US/docs/Web/Security/Secure... A true hassle for internal testing between hosts, to be honest. I just cannot run an in-development video app on my PC and connect from a phone or laptop to do some testing, without first worrying about certs at a point in development where they are superfluous and a loss of time.

akira2501 2y ago
localhost is a secure context. so.. presumably we're just waiting for .internal to be added to the white list.
- JonathonW 2y ago
  Unlikely. Localhost can be a secure context because localhost traffic doesn't leave your local machine; .internal names have no guarantees about where they go (not inconceivable that some particularly "creative" admin might have .internal names that resolve to something on the public internet).
  - Wicher 2y ago
    One can resolve "localhost" (even via an upstream resolver) to an arbitrary IP address. At least on my Linux system "localhost" only seems to be specially treated by systemd-resolved (with a cursory attempt I didn't succeed in getting it to use an upstream resolver for it). So it's not a rock-hard guarantee that traffic to localhost never leaves your system. It would be unconventional and uncommon for it to, though, except for the likes of us who like to ssh-tunnel all kinds of things on our loopback interfaces :-) The sweet spot of security vs convenience, in the case of browsers and awarding "secure origin status" for .internal, could perhaps be on a dynamic case by case basis at connect time: - check if it's using a self-signed cert - offer TOFU procedure if so - if not, verify as usual Maaaaybe check whether the connection is to an RFC1918 private range address as well. Maybe. It would break proxying and tunneling. But perhaps that'd be a good thing. This would just be for browsers, for the single purpose of enabling things like serviceworkers and other "secure origin"-only features, on this new .internal domain.

this_user 2y ago

A lot of services default to HTTPS. For instance, try setting up an internal Gitlab instance with runners, pipelines, and package/container registries that actually works. It's an absolute nightmare, and some things outright won't work. And if you want to pull images from HTTP registries with Docker, you have enable that on every instance for each registry separately. You'd be better off registering a real domain, using Let's Encrypt with the DNS challenge, and setting up an internal DNS for your services. That is literally an order of magnitude less work than setting up HTTP.

IshKebab 2y ago

A lot of modern web features now require HTTPS.

jve 2y ago

I consider HTTPS to be easier to run - you get less trouble in the end. As mentioned, some browser features are HTTPS only. You get security warnings on HTTP. Many tools now default to HTTPS by default - like newer SQL Server drivers. Dev env must resemble prod very closely so having HTTP in DEV and HTTPS in prod is asking for pain and trouble. It forces you to have some kind of expiration registry/monitoring and renewal procedures. And you happen to go throught dev env first and gain confidence and then prod. Then there are systems where client certificate is mandatory and you want to familiarize yourself already in dev/test env. Some systems even need additional configuration to allow OAuth via HTTP and that makes me feel dirty thus I rather not do it. Why do it if PROD won't have HTTP? And if one didn't know such configuration must be done, you'd be troubleshooting that system and figuring out why it doesn't work with my simple setup? Yeah, we have internal CA set up, so issuing certs are pretty easy and mostly automated and once you go HTTPS all in, you get the experience why/how things work and why they may not and got more experience to troubleshoot HTTPS stuff. You have no choice actually - the world has moved to TLS secured protocols and there is no way around getting yourself familiar with security certificates.

8organicbits 2y ago
At my first job out of college we built an API and a couple official clients for it. The testing endpoint used self-signed certs so we had to selectively configure clients to support it. Right before product launch we caught that one of our apps was ignoring certificate verification in production too due to a bug. Ever since then I've tried to run publicly valid certificates on all endpoints to eliminate those classes of bugs. I still run into accidentally disabled cert validation doing security audits, it's a common mistake.

8organicbits 2y ago

If you're on a laptop or phone that switches between WiFi networks then you are potentially spilling session cookies and other data unencrypted onto other networks that also happen to resolve .internal. HTTPS encrypts connections, but it also authenticates servers. The later is important too.