7 ms·

Firewall rules: not as secure as you think

189 points by jnord 2y ago

LinuxBender 2y ago
This methods discussed in the article are how I have seen some hardware appliance vendors SSH into their devices despite the customer only allowing outbound connections to a cloud provider. I would call these out in security reviews and it would get political really fast as the team buying the device wanted this feature but the compliance team wanted the flows documented that would conflict with security policies and ultimately would cause an audit failure. It got even more interesting when one of the vendors was also a B2B customer. A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. I will not be permitted to add specific details such as the appliance vendor or firewall vendor. I would wager some companies don't even know that a vendor can SSH into the customer datacenter not just the device despite only allowing outbound HTTPS flows to a cloud provider.
- Hizonner 2y ago
  Seriously, there were fights over that? In any remotely reasonable organization, that should be an instant, permanent blacklisting for the vendor, and termination of the "we will escort you to clean out your desk right now" variety for any internal employee who knew about it or enabled it. Probably with a line dropped to law enforcement.
  - ElevenLathe 2y ago
    That strikes me an example of how extremely unreasonable organization would behave, regardless of what it was in response to.
  - droopyEyelids 2y ago
    What reasonable company have you worked for before that had such an approach. The only place I could imagine it happening is at a boutique trading firm
- smashed 2y ago
  I understand the technical issue but on a broader sense, the instant that a vendor supplied black box is installed behind your firewall and allowed to make any sort of communication towards a vendor controlled endpoint, doesn't it immediately technically allows full remote control? Lots of talk about tunneling and wrapping/disguising ssh but a vendor does not need any of that to control its machine. For example you could have the on-prem host poll a "licensing" or "software update" server that also happens to reply with ad-hoc commands to execute on demand. Could be straight up shell commands and the result can be sent back. No need for ssh, long lived connections, reverse tunnels or anything. The only way to mitigate this is to fully trust the vendor, have a strong legal framework to protect against wrongdoings or fully block all internet access to endpoints you don't fully control.
  - LinuxBender 2y ago
    I understand where you are coming from and that it should be assumed a vendor box can do anything even without SSH. Other teams don't think this way however. "Oh its just outbound HTTPS, no inbound connections, totally fine" is the school of thought I was up against. Adding to this the box was expected to send hundreds of GB of logs to do "big-data" against daily to monitor it's health, thus training firewalls, IDS and humans to expect this outbound data volume, which could with the flip of a bit start exfiltrating customer data. If I told you what this box was for you would know how insane it would be to send more than 500KB per day for that purpose. It seems I share something in common with Boeing "line stoppers". There are details I wish I could share as they would drive home how insane this entire debacle was. I expect to read about it in the news some day.
    - Silhouette 2y ago
      This is always the problem with firewalls. If your adversary controls both ends of the connection then it's not outbound HTTPS traffic you're letting through. It's totally arbitrary two-way traffic that happens to be transmitted over a connection first established to port 443 on the remote host. The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it. In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.
- BeefySwain 2y ago
  You mention a couple times that you can't give more details. Can you say why? NDA?
  - LinuxBender 2y ago
    Multiple NDA's among other things.
- windexh8er 2y ago
  > A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. This is very easily bypassed leveraging cert pinning. Modern firewalling is all predicted on MitM approach, nobody has any secret sauce here. If they can't see inside the encryption they really can't do much. Very few customers have decryption configured correctly, or at all, at scale. Also an enterprise generally won't block connections that "aren't categorized" (URL blocklist) because it's too much work / headache Beyond that most good and bad actors have domains lying around that won't end up in blocked categories. NGFW today are NGDS (Next Gen Door Stops), they aren't effective beyond controlling their own users. And at that rate DNS is a much more cost effective control.
  - pixl97 2y ago
    >Also an enterprise generally won't block connections that "aren't categorized" Depends where. I work with a lot of large enterprise and they absolutely do block everything. Anything leaving their data centers is proxied and allow listed by the proxy. If we tried to cert pin our application, it would immediately break in their environment and would not be allowed till it passed their policies.
    - windexh8er 2y ago
      There are still many ways around this. A proxy is only as good as what the administrators have thought of as bypass. Things like domain fronting are still easily leveraged. And most organizations won't touch financial websites with a 10-foot pole because of the legal obligation of potentially decrypting PII. It's not impossible to get a domain classified as financial with a bit of work. The unfortunate reality is all it takes is one.
colemannugent 2y ago
Which is why you need a modern firewall that MitMs both TLS and SSH. Not hard to do these days.
- m463 2y ago
  Do they always work? Can't they pin certs?
  - colemannugent 2y ago
    They can pin certs, but at least you know that you can't see that traffic and make a policy decision about allowing it anyways or trying to force the vendor to drop it.
- brian_cunnie 2y ago
  I don't think a modern firewall can MiTM HTTPS TLS without triggering a "Warning: Potential Security Risk Ahead" (Firefox) or "Your connection is not private" (Chrome). Edit: typo
  - yjftsjthsd-h 2y ago
    Pretty sure you still can, it just requires that the client system trusts the CA being used to sign the MITM certs. That obviously limits the cases where it works, but not to zero.
    - ronsor 2y ago
      Because this has been abused, a lot of (mobile) apps use certificate pinning and will not accept MITM, even with a custom CA installed.
      - yjftsjthsd-h 2y ago
        I don't for a moment believe that that's the reason (more likely, it's the apps trying to prevent reverse engineering), but yes, there's a bit of a cat/mouse game where you can read traffic but HTTPS prevents that but you can add a custom CA but apps can pin certs but you can modify the app to fix that. But I suspect that for the appliance case, a business can just require that the vendor allow a custom CA and block any traffic they can't decrypt.
taeric 2y ago
This is basically a ridiculous arms race between the people making rules and people that feel they are an exception to the rule. Can really suck when the reasoning on both ends is valid
- scrapheap 2y ago
  The worst bit is where it's happening because neither side wants to just sit down discuss what's actually required and how to provide it securely.
  - pixl97 2y ago
    "neither side" This is not a coin, it's more like an octahedron. You have the end user. You have the person at the company managing the application. You have networking team. You have the edge and firewall teams. You have the security team. You have a compliance team. You have the upper management looking at controlling costs.
pid-1 2y ago
Most security controls used in your average business can be bypassed by knowledgeable users with enough time. The aim is to make things as hard as reasonably possible so you can tell your boss and regulators that you did your part.
- mschuster91 2y ago
  The problem is, most organizations - particularly large ones, but following the advent of "cyber insurances" also more and more smaller ones - drown in byzantine bureaucracy and requirements that makes work excessively difficult. Any organization depends on people willing to bend, stretch and bypass the rules where necessary - refusing to do so is considered to be a form of labor action [1]. [1] https://en.wikipedia.org/wiki/Work-to-rule https://en.wikipedia.org/wiki/Work-to-rule
nunez 2y ago
> You (the vendor) can ask the customer to open their firewall for your software to communicate with the outside world (e.g. your own datacenter or third party services), but customers will usually be reluctant to open their firewall more than necessary. Vendors that try tricks like this to work around the firewall team, as backlogged and inefficient as they are, will lose deals and credibility, 100% guaranteed. And I love ssh -R and stunnel, for what it's worth. The easier way to get what you want is to get someone technical with political capital (they exist) to buy into what you're selling so that they can tell the firewall team to make exceptions during a trial period.
evulhotdog 2y ago
I’ve had these types of conversations hundreds of times due to some architecture of various systems that I’ve worked on over the years. Mostly from the vendor side. The only real way to combat this is to do Deep Packet Inspection (DPI) and then to look at all the data that’s being passed within that encrypted connection. The problem with this, is that at this point, the vendor has to trust that the customer is doing their due diligence to protect anything that they find within that associated connection with the same diligence that the vendor would be. As a vendor specifically in the healthcare space, I can tell you that there is no way in hell that I am going to trust any of our customers to secure our data, more than ourselves. They will never know or understand all of the components and know where the risks lie better than the vendor. For example, in the infancy of one of the companies that I worked at, I agreed with one of our customers to allow certificate pinning, and we would install their certificate on our servers which would allow them to inspect the traffic. Wouldn’t you know it, there was an issue where they were blocking some of our traffic that needed to go outbound, because of their deep packet inspection triggered some rule that they had enabled. Conveniently while they were pilfering through the data to troubleshoot the issue, they sent a bunch of the payloads that contained various API keys, tokens, etc. which were now just out there in the wild, that under our watch would never see the light of day. Who knows where else those things are logged or what other places besides the 30 or so recipients that were on that email thread. As soon as we found out that they were not handling it appropriately, we took corrective actions, not only to replace the keys, but also to disallow that going forward. And this for context, is one of the biggest healthcare institutions in the United States. I can confidently say that I have a strong security mindset and anything that gets built has security at the forefront of every release. You can’t trust people who aren’t liable for your systems, with your data, or even to protect their own data. Maybe I am jaded, but the lesson I learned is that you shouldn’t trust anybody, and that chances are other people will not treat sensitive information with the same sensitivity that you will.
- Spooky23 2y ago
  In that sort of scenario, the appliance needs to be in a DMZ and treated like an external system. Personally, this is why I hate systems that need this sort of connectivity beyond a SAN service processor or similar. I’d rather have the third party just run in on their premise with appropriate contracts than pretend it’s just another server in the datacenter.
nirui 2y ago
For the SSH case mentioned in the article, `ssh -R` trick should already resolve some one-time contingencies (assuming SSH connection is not blocked). But if you find yourself requesting `ssh -R` too often, maybe just ask those datacenter people to setup a proper SSH Bastion for you. There are opensource solutions and enterprise-level ones (Teleport for example: https://github.com/gravitational/teleport https://github.com/gravitational/teleport), some also allows you to do audit and access control, which maybe important if you work for a enterprise client. The DIY solution described in the article literally punched a hole in the firewall. The firewall people might not like it.
- karmarepellent 2y ago
  Regarding SSH bastion hosts, apart from open-source and enterprise solutions that may add some valuable features, you can always get away with a properly configured SSH jump host using TcpForwarding to relay connections to the target host.
  - nirui 2y ago
    There is a problem on simple TcpForwarding: audition. Some SSH Bastion allows all input and output to be logged for later inspection and forensic while DIY solutions often just ignored it's importance. Notice that I said "forensic"? Yeah, assuming some very unfortunate situation has occurred, police are called and there will be a full incident report on it, a DIY'ed apparatus without detailed logging records will make the deployer of that apparatus look really bad. This is especially true if you're a contractor, since your hirer might not fully trust you. Also, at very least, you'll probably want to be mentioned as "the proficient and diligent from one of our employee/partner allowed us to collect detailed information on what the attacker has done on our systems" than "the lack of consideration on the use of remote access tool made it impossible for us to know what else the attacker has done". It's good for your resume.
    - karmarepellent 2y ago
      Thorough auditing can still happen on the target host. If every single one of your hosts is properly configured to produce audit logs, maybe you can get away without auditing or even session recording on the SSH bastion host.
tptacek 2y ago
This is billed as a means of selling customers on-prem stuff that you can remote-manage into with SSH despite firewall rules blocking SSH. You can do this. You can get a lot more sophisticated than the tricks outlined in this article to make it happen. It is very difficult for customers to prevent you from doing it. And, if you do it, you're going to get famous for doing it, when a customer that actually cares about your network security notices that you built a remote tunnel into their network. I strongly advise anyone making product decisions to assume that none of these tricks work, and that there are no tricks you can use to build discreet remote management tunnels to devices (including hosts running your software) that have customer internal addresses assigned.
- spydum 2y ago
  I've seen vendors offering this technique or similar, but making it "opt-in". For example, Okta Access Gateway used to perform a reverse tunnel out to an Okta managed IP, but you had to enable the "Support VPN" option on the device. https://help.okta.com/oag/en-us/content/topics/access-gateway/about-oag-support-vpn.htm https://help.okta.com/oag/en-us/content/topics/access-gatewa... Seems like they dropped the feature, not sure if from customer backlash, or their security engineering teams finally realizing that it's risky. However, it was at least documented, and customer toggleable.
- mst 2y ago
  I think that were I to implement anything like this I would document the capability explicitly. The situation that seems useful to me is bypassing dysfunctional processes rather than circumventing inconvenient policies. (and if the device in question can auto-apply updates so far as I can see being able to ssh into it rather than ship it an update that Does Something is more a question of how convenient it is to Do The Thing rather than adding any additional Things that it Can Do, though it's entirely possible I'm missing something important there)
CraigJPerry 2y ago
Corkscrew, ohh that takes me back to the time I nearly got sacked. The GitHub repo is 8 years old but corkscrew is much older. I was working at a big place at the time. I had a fairly extensive home lab. I used to practice data migrations on synthetic data at home, provoke failures in my safe home lab env then write scripts to automate the migration while catching all the gotchas I could think of (think disk space filling up mid-migration, that kinda thing). Anyway, I was using ssh pre commands and netcat (corkscrew makes this way simpler to do, I didn’t learn about corkscrew until afterwards) to punch through from my desk at work to my lab at home so I could copy in my scripts. Big no-no but at the time that was not clear to me at all. I didn’t even consider I might be violating a rule. They flew a couple of “security officers” to the office I worked at to give me the full shake down… ooops! A frustrating thing was that the people they flew (at great expense, international flight) were not that technical, one was an ex-cop as I recall. Trying to explain what I was doing and how it worked was pretty tough! Actually the most frustrating thing from my point of view was in my mind I was showing initiative and doing all this free work outside of work and here I was getting threatened with dismissal…
- znpy 2y ago
  I didn't know the tool and it looks, interesting, I'll give it a try
- mrmetanoia 2y ago
  Amusing that you patronize corporate security for not totally understanding what you were doing, but couldn't deduce tunneling home to grab your data exfiltration - i mean migration - scripts might not be kosher with security.
PhilippGille 2y ago
The article says that Squid can only do HTTP, and suggests to add a reverse proxy in front of Squid for HTTPS, but doesn't Squid support HTTPS itself via "SSL bump"?
donatj 2y ago
We basically have tunnel just to get our work done. We host a number of services in China, serving the Chinese market, and our corporate firewall blocks our own access to them despite numerous requests to IT to resolve the issue. We just use SSH to bounce off one of our ec2 instance to work on these.
xrd 2y ago
This makes me grimace. I recall trying to install an SSL cert inside a very restricted environment. I had to build a special mode into our application so the IT person could use cURL to hit an endpoint with the base64'ed cert, and then scan the log files for the chunks and then reassemble them by hand with vi. Imagine coaching someone to use VI and bash and grep who only knows how to click around in Windows, everyone feels like a complete idiot. They completely disabled all ports other than HTTPS, did not allow copy-paste into the VM. Inbound HTTP was disabled (so no wget, can't even install extra software with yum). What a nightmare. The irony of these locked down environments is that they are put in place by crusty IT people who have good intentions, but also know it helps them keep their jobs. But, because of it, all the on-premise software is moving to the cloud, and those people are going to lose their jobs anyway. This was a sad realization I found after years of enterprise sales work. I thought on-premise software could give us an advantage because as a small company you can offer a more bespoke experience. But, we always looked bad because everyone fights to protect their fiefdom, and our software was blocked at every turn. It is crazy that enterprises are moving everything to the cloud and no one is noticing that the barrier to getting access to their private sensitive data is more or less water vapor, so to speak. That's nuts.
JackSlateur 2y ago
Firewall. Because why fight hostile actors when you can just fight your teammates ? Firewall and people behind them are actively hostile to the compagny. Those are relics from decades ago, when people could map the entire internet on their devices. In 2024, this is nothing but a clown circus. They try to reconciliate an ever-changing world with a never-changing world. So they make exceptions, thousands of exceptions, everything becomes an exception. And then, they think : hey, we are doing L3/L4, this is the issue ! We fail because we are not L7. And the circus comes around: corporate TLS mitm. Massive project, custom certificates must be deployed in all and every compagny devices. Thus, exceptions, again more exceptions : what about this device ? We cannot add our certs here. Exception. Ha, this specific stuff cannot be mitmed (maybe they implemented certificate pinning ? good guys). No problem : exceptions ! On top on that, all this cruft is expensive as hell. So, we add more exceptions for stuff that is deemed "secure enough" : google meet / zoom / whatever. Various objects storages (s3 && friends). More exceptions. At the end, you spent millions, ate thousands of FTE on the project. To build a massive amount of exceptions (which basically allows everything, indeed). The worst is this: for every exception, you have someone, who wants to work for the compagny, who is blocked from doing this work, who have to wait, argue and beg to finally be allowed to work for the compagny. (source: experience, I'm a network architect, worked for a couple of multi-billions-$ compagnies)
- jghxox 2y ago
  [flagged]
- transpute 2y ago
  > Firewall ... And then, they think : hey, we are doing L3/L4, this is the issue ! We fail because we are not L7. Outside of corporate firewalls, these fractals reappear at the scale of nation-state firewalls. What do you think of "zero-trust" and "software-defined perimeter" approaches where every network connection is linked to identity and risk assessment?
- marcosdumay 2y ago
  > exceptions (which basically allows everything, indeed) No, that's not right. You've allowed everything since the beginning. All the exceptions are for honest software, nothing malicious need it.
gostsamo 2y ago
This is made to sound much more malicious than it is. Sometimes this techniques are used just because you don't want your product to have opened ports which could be potential entry points for uninvited guests. As far as the client knows about the connection and what it entails, it should be okay. The product of my employer is remote monitoring and management so the nature of the product is to allow cloud visibility to a network and the reverse connection is actually improving security.
Attummm 2y ago
Just so that we are all on the same page, that is illegal and unethical A customer might have demands which are opposing; that doesn't mean you should hack your customers with backdoors. These backdoors can also be used by others. Although its neat trick that doesn't mean it's ethical or legal nor desirable. Think of this what if each application/service you use would create backdoors to your servers/devices.
deleted 2y ago
[deleted]
nprateem 2y ago
I don't think repeatedly referring to gaining unauthorised access to a network as a "trick" would trick a judge (or corporate lawyers) much at all...
jerf 2y ago
I can tell you that at scale, customers will notice this. They notice what DNS addresses you're hitting. They notice what outbound sockets you open. They notice the traffic patterns don't match the protocol you're putatively using. A given customer most likely will not, but there are networks out there where they're looking at pretty much every connection, and it only takes one bad interaction with one customer going to one press outlet for you to have a Bad Day. It's bad enough when the customer misunderstood something and you've got a good answer as to what is actually going on. You really don't want to be trying to do business in a scenario where you really were doing something like this.
perryizgr8 2y ago
Doesn't tailscale also do similar thing where you never open any port in your server but you can still SSH into it as long as the daemon is running on the server?
mmsc 2y ago
This article talks about using squid. I wouldn't recommend Squid for this, as they're understaffed and took years to fix critical vulnerabilities I found[0]. Using ssh-over-https with ssh -R works wonders everywhere though. You could probably even make the ssh packets look like html so it's opaque to a mitm-proxy too. The sort of firewalls this post is discussing are close to snake oil imo. Sure they help with automated script kiddie attacks and whatnot, but yes, if you control both ends, it's nearly always possible to connect back. 0: https://megamansec.github.io/Squid-Security-Audit/ https://megamansec.github.io/Squid-Security-Audit/

189 points by jnord 2y ago

This methods discussed in the article are how I have seen some hardware appliance vendors SSH into their devices despite the customer only allowing outbound connections to a cloud provider. I would call these out in security reviews and it would get political really fast as the team buying the device wanted this feature but the compliance team wanted the flows documented that would conflict with security policies and ultimately would cause an audit failure. It got even more interesting when one of the vendors was also a B2B customer. A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. I will not be permitted to add specific details such as the appliance vendor or firewall vendor. I would wager some companies don't even know that a vendor can SSH into the customer datacenter not just the device despite only allowing outbound HTTPS flows to a cloud provider.

Hizonner 2y ago
Seriously, there were fights over that? In any remotely reasonable organization, that should be an instant, permanent blacklisting for the vendor, and termination of the "we will escort you to clean out your desk right now" variety for any internal employee who knew about it or enabled it. Probably with a line dropped to law enforcement.
- ElevenLathe 2y ago
  That strikes me an example of how extremely unreasonable organization would behave, regardless of what it was in response to.
- droopyEyelids 2y ago
  What reasonable company have you worked for before that had such an approach. The only place I could imagine it happening is at a boutique trading firm
smashed 2y ago
I understand the technical issue but on a broader sense, the instant that a vendor supplied black box is installed behind your firewall and allowed to make any sort of communication towards a vendor controlled endpoint, doesn't it immediately technically allows full remote control? Lots of talk about tunneling and wrapping/disguising ssh but a vendor does not need any of that to control its machine. For example you could have the on-prem host poll a "licensing" or "software update" server that also happens to reply with ad-hoc commands to execute on demand. Could be straight up shell commands and the result can be sent back. No need for ssh, long lived connections, reverse tunnels or anything. The only way to mitigate this is to fully trust the vendor, have a strong legal framework to protect against wrongdoings or fully block all internet access to endpoints you don't fully control.
- LinuxBender 2y ago
  I understand where you are coming from and that it should be assumed a vendor box can do anything even without SSH. Other teams don't think this way however. "Oh its just outbound HTTPS, no inbound connections, totally fine" is the school of thought I was up against. Adding to this the box was expected to send hundreds of GB of logs to do "big-data" against daily to monitor it's health, thus training firewalls, IDS and humans to expect this outbound data volume, which could with the flip of a bit start exfiltrating customer data. If I told you what this box was for you would know how insane it would be to send more than 500KB per day for that purpose. It seems I share something in common with Boeing "line stoppers". There are details I wish I could share as they would drive home how insane this entire debacle was. I expect to read about it in the news some day.
  - Silhouette 2y ago
    This is always the problem with firewalls. If your adversary controls both ends of the connection then it's not outbound HTTPS traffic you're letting through. It's totally arbitrary two-way traffic that happens to be transmitted over a connection first established to port 443 on the remote host. The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it. In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.
BeefySwain 2y ago
You mention a couple times that you can't give more details. Can you say why? NDA?
- LinuxBender 2y ago
  Multiple NDA's among other things.
windexh8er 2y ago
> A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. This is very easily bypassed leveraging cert pinning. Modern firewalling is all predicted on MitM approach, nobody has any secret sauce here. If they can't see inside the encryption they really can't do much. Very few customers have decryption configured correctly, or at all, at scale. Also an enterprise generally won't block connections that "aren't categorized" (URL blocklist) because it's too much work / headache Beyond that most good and bad actors have domains lying around that won't end up in blocked categories. NGFW today are NGDS (Next Gen Door Stops), they aren't effective beyond controlling their own users. And at that rate DNS is a much more cost effective control.
- pixl97 2y ago
  >Also an enterprise generally won't block connections that "aren't categorized" Depends where. I work with a lot of large enterprise and they absolutely do block everything. Anything leaving their data centers is proxied and allow listed by the proxy. If we tried to cert pin our application, it would immediately break in their environment and would not be allowed till it passed their policies.
  - windexh8er 2y ago
    There are still many ways around this. A proxy is only as good as what the administrators have thought of as bypass. Things like domain fronting are still easily leveraged. And most organizations won't touch financial websites with a 10-foot pole because of the legal obligation of potentially decrypting PII. It's not impossible to get a domain classified as financial with a bit of work. The unfortunate reality is all it takes is one.

colemannugent 2y ago

Which is why you need a modern firewall that MitMs both TLS and SSH. Not hard to do these days.

m463 2y ago
Do they always work? Can't they pin certs?
- colemannugent 2y ago
  They can pin certs, but at least you know that you can't see that traffic and make a policy decision about allowing it anyways or trying to force the vendor to drop it.
brian_cunnie 2y ago
I don't think a modern firewall can MiTM HTTPS TLS without triggering a "Warning: Potential Security Risk Ahead" (Firefox) or "Your connection is not private" (Chrome). Edit: typo
- yjftsjthsd-h 2y ago
  Pretty sure you still can, it just requires that the client system trusts the CA being used to sign the MITM certs. That obviously limits the cases where it works, but not to zero.
  - ronsor 2y ago
    Because this has been abused, a lot of (mobile) apps use certificate pinning and will not accept MITM, even with a custom CA installed.
    - yjftsjthsd-h 2y ago
      I don't for a moment believe that that's the reason (more likely, it's the apps trying to prevent reverse engineering), but yes, there's a bit of a cat/mouse game where you can read traffic but HTTPS prevents that but you can add a custom CA but apps can pin certs but you can modify the app to fix that. But I suspect that for the appliance case, a business can just require that the vendor allow a custom CA and block any traffic they can't decrypt.

taeric 2y ago

This is basically a ridiculous arms race between the people making rules and people that feel they are an exception to the rule. Can really suck when the reasoning on both ends is valid

scrapheap 2y ago
The worst bit is where it's happening because neither side wants to just sit down discuss what's actually required and how to provide it securely.
- pixl97 2y ago
  "neither side" This is not a coin, it's more like an octahedron. You have the end user. You have the person at the company managing the application. You have networking team. You have the edge and firewall teams. You have the security team. You have a compliance team. You have the upper management looking at controlling costs.

pid-1 2y ago

Most security controls used in your average business can be bypassed by knowledgeable users with enough time. The aim is to make things as hard as reasonably possible so you can tell your boss and regulators that you did your part.

mschuster91 2y ago
The problem is, most organizations - particularly large ones, but following the advent of "cyber insurances" also more and more smaller ones - drown in byzantine bureaucracy and requirements that makes work excessively difficult. Any organization depends on people willing to bend, stretch and bypass the rules where necessary - refusing to do so is considered to be a form of labor action [1]. [1] https://en.wikipedia.org/wiki/Work-to-rule https://en.wikipedia.org/wiki/Work-to-rule

nunez 2y ago

> You (the vendor) can ask the customer to open their firewall for your software to communicate with the outside world (e.g. your own datacenter or third party services), but customers will usually be reluctant to open their firewall more than necessary. Vendors that try tricks like this to work around the firewall team, as backlogged and inefficient as they are, will lose deals and credibility, 100% guaranteed. And I love ssh -R and stunnel, for what it's worth. The easier way to get what you want is to get someone technical with political capital (they exist) to buy into what you're selling so that they can tell the firewall team to make exceptions during a trial period.

evulhotdog 2y ago

I’ve had these types of conversations hundreds of times due to some architecture of various systems that I’ve worked on over the years. Mostly from the vendor side. The only real way to combat this is to do Deep Packet Inspection (DPI) and then to look at all the data that’s being passed within that encrypted connection. The problem with this, is that at this point, the vendor has to trust that the customer is doing their due diligence to protect anything that they find within that associated connection with the same diligence that the vendor would be. As a vendor specifically in the healthcare space, I can tell you that there is no way in hell that I am going to trust any of our customers to secure our data, more than ourselves. They will never know or understand all of the components and know where the risks lie better than the vendor. For example, in the infancy of one of the companies that I worked at, I agreed with one of our customers to allow certificate pinning, and we would install their certificate on our servers which would allow them to inspect the traffic. Wouldn’t you know it, there was an issue where they were blocking some of our traffic that needed to go outbound, because of their deep packet inspection triggered some rule that they had enabled. Conveniently while they were pilfering through the data to troubleshoot the issue, they sent a bunch of the payloads that contained various API keys, tokens, etc. which were now just out there in the wild, that under our watch would never see the light of day. Who knows where else those things are logged or what other places besides the 30 or so recipients that were on that email thread. As soon as we found out that they were not handling it appropriately, we took corrective actions, not only to replace the keys, but also to disallow that going forward. And this for context, is one of the biggest healthcare institutions in the United States. I can confidently say that I have a strong security mindset and anything that gets built has security at the forefront of every release. You can’t trust people who aren’t liable for your systems, with your data, or even to protect their own data. Maybe I am jaded, but the lesson I learned is that you shouldn’t trust anybody, and that chances are other people will not treat sensitive information with the same sensitivity that you will.

Spooky23 2y ago
In that sort of scenario, the appliance needs to be in a DMZ and treated like an external system. Personally, this is why I hate systems that need this sort of connectivity beyond a SAN service processor or similar. I’d rather have the third party just run in on their premise with appropriate contracts than pretend it’s just another server in the datacenter.

nirui 2y ago

For the SSH case mentioned in the article, `ssh -R` trick should already resolve some one-time contingencies (assuming SSH connection is not blocked). But if you find yourself requesting `ssh -R` too often, maybe just ask those datacenter people to setup a proper SSH Bastion for you. There are opensource solutions and enterprise-level ones (Teleport for example: https://github.com/gravitational/teleport https://github.com/gravitational/teleport), some also allows you to do audit and access control, which maybe important if you work for a enterprise client. The DIY solution described in the article literally punched a hole in the firewall. The firewall people might not like it.

karmarepellent 2y ago
Regarding SSH bastion hosts, apart from open-source and enterprise solutions that may add some valuable features, you can always get away with a properly configured SSH jump host using TcpForwarding to relay connections to the target host.
- nirui 2y ago
  There is a problem on simple TcpForwarding: audition. Some SSH Bastion allows all input and output to be logged for later inspection and forensic while DIY solutions often just ignored it's importance. Notice that I said "forensic"? Yeah, assuming some very unfortunate situation has occurred, police are called and there will be a full incident report on it, a DIY'ed apparatus without detailed logging records will make the deployer of that apparatus look really bad. This is especially true if you're a contractor, since your hirer might not fully trust you. Also, at very least, you'll probably want to be mentioned as "the proficient and diligent from one of our employee/partner allowed us to collect detailed information on what the attacker has done on our systems" than "the lack of consideration on the use of remote access tool made it impossible for us to know what else the attacker has done". It's good for your resume.
  - karmarepellent 2y ago
    Thorough auditing can still happen on the target host. If every single one of your hosts is properly configured to produce audit logs, maybe you can get away without auditing or even session recording on the SSH bastion host.

tptacek 2y ago

This is billed as a means of selling customers on-prem stuff that you can remote-manage into with SSH despite firewall rules blocking SSH. You can do this. You can get a lot more sophisticated than the tricks outlined in this article to make it happen. It is very difficult for customers to prevent you from doing it. And, if you do it, you're going to get famous for doing it, when a customer that actually cares about your network security notices that you built a remote tunnel into their network. I strongly advise anyone making product decisions to assume that none of these tricks work, and that there are no tricks you can use to build discreet remote management tunnels to devices (including hosts running your software) that have customer internal addresses assigned.

spydum 2y ago
I've seen vendors offering this technique or similar, but making it "opt-in". For example, Okta Access Gateway used to perform a reverse tunnel out to an Okta managed IP, but you had to enable the "Support VPN" option on the device. https://help.okta.com/oag/en-us/content/topics/access-gateway/about-oag-support-vpn.htm https://help.okta.com/oag/en-us/content/topics/access-gatewa... Seems like they dropped the feature, not sure if from customer backlash, or their security engineering teams finally realizing that it's risky. However, it was at least documented, and customer toggleable.
mst 2y ago
I think that were I to implement anything like this I would document the capability explicitly. The situation that seems useful to me is bypassing dysfunctional processes rather than circumventing inconvenient policies. (and if the device in question can auto-apply updates so far as I can see being able to ssh into it rather than ship it an update that Does Something is more a question of how convenient it is to Do The Thing rather than adding any additional Things that it Can Do, though it's entirely possible I'm missing something important there)

CraigJPerry 2y ago

Corkscrew, ohh that takes me back to the time I nearly got sacked. The GitHub repo is 8 years old but corkscrew is much older. I was working at a big place at the time. I had a fairly extensive home lab. I used to practice data migrations on synthetic data at home, provoke failures in my safe home lab env then write scripts to automate the migration while catching all the gotchas I could think of (think disk space filling up mid-migration, that kinda thing). Anyway, I was using ssh pre commands and netcat (corkscrew makes this way simpler to do, I didn’t learn about corkscrew until afterwards) to punch through from my desk at work to my lab at home so I could copy in my scripts. Big no-no but at the time that was not clear to me at all. I didn’t even consider I might be violating a rule. They flew a couple of “security officers” to the office I worked at to give me the full shake down… ooops! A frustrating thing was that the people they flew (at great expense, international flight) were not that technical, one was an ex-cop as I recall. Trying to explain what I was doing and how it worked was pretty tough! Actually the most frustrating thing from my point of view was in my mind I was showing initiative and doing all this free work outside of work and here I was getting threatened with dismissal…

znpy 2y ago
I didn't know the tool and it looks, interesting, I'll give it a try
mrmetanoia 2y ago
Amusing that you patronize corporate security for not totally understanding what you were doing, but couldn't deduce tunneling home to grab your data exfiltration - i mean migration - scripts might not be kosher with security.

PhilippGille 2y ago

The article says that Squid can only do HTTP, and suggests to add a reverse proxy in front of Squid for HTTPS, but doesn't Squid support HTTPS itself via "SSL bump"?

donatj 2y ago

We basically have tunnel just to get our work done. We host a number of services in China, serving the Chinese market, and our corporate firewall blocks our own access to them despite numerous requests to IT to resolve the issue. We just use SSH to bounce off one of our ec2 instance to work on these.

xrd 2y ago

This makes me grimace. I recall trying to install an SSL cert inside a very restricted environment. I had to build a special mode into our application so the IT person could use cURL to hit an endpoint with the base64'ed cert, and then scan the log files for the chunks and then reassemble them by hand with vi. Imagine coaching someone to use VI and bash and grep who only knows how to click around in Windows, everyone feels like a complete idiot. They completely disabled all ports other than HTTPS, did not allow copy-paste into the VM. Inbound HTTP was disabled (so no wget, can't even install extra software with yum). What a nightmare. The irony of these locked down environments is that they are put in place by crusty IT people who have good intentions, but also know it helps them keep their jobs. But, because of it, all the on-premise software is moving to the cloud, and those people are going to lose their jobs anyway. This was a sad realization I found after years of enterprise sales work. I thought on-premise software could give us an advantage because as a small company you can offer a more bespoke experience. But, we always looked bad because everyone fights to protect their fiefdom, and our software was blocked at every turn. It is crazy that enterprises are moving everything to the cloud and no one is noticing that the barrier to getting access to their private sensitive data is more or less water vapor, so to speak. That's nuts.

JackSlateur 2y ago

Firewall. Because why fight hostile actors when you can just fight your teammates ? Firewall and people behind them are actively hostile to the compagny. Those are relics from decades ago, when people could map the entire internet on their devices. In 2024, this is nothing but a clown circus. They try to reconciliate an ever-changing world with a never-changing world. So they make exceptions, thousands of exceptions, everything becomes an exception. And then, they think : hey, we are doing L3/L4, this is the issue ! We fail because we are not L7. And the circus comes around: corporate TLS mitm. Massive project, custom certificates must be deployed in all and every compagny devices. Thus, exceptions, again more exceptions : what about this device ? We cannot add our certs here. Exception. Ha, this specific stuff cannot be mitmed (maybe they implemented certificate pinning ? good guys). No problem : exceptions ! On top on that, all this cruft is expensive as hell. So, we add more exceptions for stuff that is deemed "secure enough" : google meet / zoom / whatever. Various objects storages (s3 && friends). More exceptions. At the end, you spent millions, ate thousands of FTE on the project. To build a massive amount of exceptions (which basically allows everything, indeed). The worst is this: for every exception, you have someone, who wants to work for the compagny, who is blocked from doing this work, who have to wait, argue and beg to finally be allowed to work for the compagny. (source: experience, I'm a network architect, worked for a couple of multi-billions-$ compagnies)

jghxox 2y ago
[flagged]
transpute 2y ago
> Firewall ... And then, they think : hey, we are doing L3/L4, this is the issue ! We fail because we are not L7. Outside of corporate firewalls, these fractals reappear at the scale of nation-state firewalls. What do you think of "zero-trust" and "software-defined perimeter" approaches where every network connection is linked to identity and risk assessment?
marcosdumay 2y ago
> exceptions (which basically allows everything, indeed) No, that's not right. You've allowed everything since the beginning. All the exceptions are for honest software, nothing malicious need it.

gostsamo 2y ago

This is made to sound much more malicious than it is. Sometimes this techniques are used just because you don't want your product to have opened ports which could be potential entry points for uninvited guests. As far as the client knows about the connection and what it entails, it should be okay. The product of my employer is remote monitoring and management so the nature of the product is to allow cloud visibility to a network and the reverse connection is actually improving security.

Attummm 2y ago

Just so that we are all on the same page, that is illegal and unethical A customer might have demands which are opposing; that doesn't mean you should hack your customers with backdoors. These backdoors can also be used by others. Although its neat trick that doesn't mean it's ethical or legal nor desirable. Think of this what if each application/service you use would create backdoors to your servers/devices.

deleted 2y ago

[deleted]

nprateem 2y ago

I don't think repeatedly referring to gaining unauthorised access to a network as a "trick" would trick a judge (or corporate lawyers) much at all...

jerf 2y ago

I can tell you that at scale, customers will notice this. They notice what DNS addresses you're hitting. They notice what outbound sockets you open. They notice the traffic patterns don't match the protocol you're putatively using. A given customer most likely will not, but there are networks out there where they're looking at pretty much every connection, and it only takes one bad interaction with one customer going to one press outlet for you to have a Bad Day. It's bad enough when the customer misunderstood something and you've got a good answer as to what is actually going on. You really don't want to be trying to do business in a scenario where you really were doing something like this.

perryizgr8 2y ago

Doesn't tailscale also do similar thing where you never open any port in your server but you can still SSH into it as long as the daemon is running on the server?

mmsc 2y ago

This article talks about using squid. I wouldn't recommend Squid for this, as they're understaffed and took years to fix critical vulnerabilities I found[0]. Using ssh-over-https with ssh -R works wonders everywhere though. You could probably even make the ssh packets look like html so it's opaque to a mitm-proxy too. The sort of firewalls this post is discussing are close to snake oil imo. Sure they help with automated script kiddie attacks and whatnot, but yes, if you control both ends, it's nearly always possible to connect back. 0: https://megamansec.github.io/Squid-Security-Audit/ https://megamansec.github.io/Squid-Security-Audit/