5 ms·

This methods discussed in the article are how I have seen some hardware appliance vendors SSH into their devices despite the customer only allowing outbound con

This methods discussed in the article are how I have seen some hardware appliance vendors SSH into their devices despite the customer only allowing outbound connections to a cloud provider. I would call these out in security reviews and it would get political really fast as the team buying the device wanted this feature but the compliance team wanted the flows documented that would conflict with security policies and ultimately would cause an audit failure. It got even more interesting when one of the vendors was also a B2B customer. A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. I will not be permitted to add specific details such as the appliance vendor or firewall vendor. I would wager some companies don't even know that a vendor can SSH into the customer datacenter not just the device despite only allowing outbound HTTPS flows to a cloud provider.

Hizonner 2y ago
Seriously, there were fights over that? In any remotely reasonable organization, that should be an instant, permanent blacklisting for the vendor, and termination of the "we will escort you to clean out your desk right now" variety for any internal employee who knew about it or enabled it. Probably with a line dropped to law enforcement.
- ElevenLathe 2y ago
  That strikes me an example of how extremely unreasonable organization would behave, regardless of what it was in response to.
- droopyEyelids 2y ago
  What reasonable company have you worked for before that had such an approach. The only place I could imagine it happening is at a boutique trading firm
smashed 2y ago
I understand the technical issue but on a broader sense, the instant that a vendor supplied black box is installed behind your firewall and allowed to make any sort of communication towards a vendor controlled endpoint, doesn't it immediately technically allows full remote control? Lots of talk about tunneling and wrapping/disguising ssh but a vendor does not need any of that to control its machine. For example you could have the on-prem host poll a "licensing" or "software update" server that also happens to reply with ad-hoc commands to execute on demand. Could be straight up shell commands and the result can be sent back. No need for ssh, long lived connections, reverse tunnels or anything. The only way to mitigate this is to fully trust the vendor, have a strong legal framework to protect against wrongdoings or fully block all internet access to endpoints you don't fully control.
- LinuxBender 2y ago
  I understand where you are coming from and that it should be assumed a vendor box can do anything even without SSH. Other teams don't think this way however. "Oh its just outbound HTTPS, no inbound connections, totally fine" is the school of thought I was up against. Adding to this the box was expected to send hundreds of GB of logs to do "big-data" against daily to monitor it's health, thus training firewalls, IDS and humans to expect this outbound data volume, which could with the flip of a bit start exfiltrating customer data. If I told you what this box was for you would know how insane it would be to send more than 500KB per day for that purpose. It seems I share something in common with Boeing "line stoppers". There are details I wish I could share as they would drive home how insane this entire debacle was. I expect to read about it in the news some day.
  - Silhouette 2y ago
    This is always the problem with firewalls. If your adversary controls both ends of the connection then it's not outbound HTTPS traffic you're letting through. It's totally arbitrary two-way traffic that happens to be transmitted over a connection first established to port 443 on the remote host. The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it. In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.
  - j-bos 2y ago
    Sounds like a situation needing a whistleblower.
    - BonoboIO 2y ago
BeefySwain 2y ago
You mention a couple times that you can't give more details. Can you say why? NDA?
- LinuxBender 2y ago
  Multiple NDA's among other things.
windexh8er 2y ago
> A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. This is very easily bypassed leveraging cert pinning. Modern firewalling is all predicted on MitM approach, nobody has any secret sauce here. If they can't see inside the encryption they really can't do much. Very few customers have decryption configured correctly, or at all, at scale. Also an enterprise generally won't block connections that "aren't categorized" (URL blocklist) because it's too much work / headache Beyond that most good and bad actors have domains lying around that won't end up in blocked categories. NGFW today are NGDS (Next Gen Door Stops), they aren't effective beyond controlling their own users. And at that rate DNS is a much more cost effective control.
- pixl97 2y ago
  >Also an enterprise generally won't block connections that "aren't categorized" Depends where. I work with a lot of large enterprise and they absolutely do block everything. Anything leaving their data centers is proxied and allow listed by the proxy. If we tried to cert pin our application, it would immediately break in their environment and would not be allowed till it passed their policies.
  - windexh8er 2y ago
    There are still many ways around this. A proxy is only as good as what the administrators have thought of as bypass. Things like domain fronting are still easily leveraged. And most organizations won't touch financial websites with a 10-foot pole because of the legal obligation of potentially decrypting PII. It's not impossible to get a domain classified as financial with a bit of work. The unfortunate reality is all it takes is one.

Hizonner 2y ago

Seriously, there were fights over that? In any remotely reasonable organization, that should be an instant, permanent blacklisting for the vendor, and termination of the "we will escort you to clean out your desk right now" variety for any internal employee who knew about it or enabled it. Probably with a line dropped to law enforcement.

ElevenLathe 2y ago
That strikes me an example of how extremely unreasonable organization would behave, regardless of what it was in response to.
droopyEyelids 2y ago
What reasonable company have you worked for before that had such an approach. The only place I could imagine it happening is at a boutique trading firm

smashed 2y ago

I understand the technical issue but on a broader sense, the instant that a vendor supplied black box is installed behind your firewall and allowed to make any sort of communication towards a vendor controlled endpoint, doesn't it immediately technically allows full remote control? Lots of talk about tunneling and wrapping/disguising ssh but a vendor does not need any of that to control its machine. For example you could have the on-prem host poll a "licensing" or "software update" server that also happens to reply with ad-hoc commands to execute on demand. Could be straight up shell commands and the result can be sent back. No need for ssh, long lived connections, reverse tunnels or anything. The only way to mitigate this is to fully trust the vendor, have a strong legal framework to protect against wrongdoings or fully block all internet access to endpoints you don't fully control.

LinuxBender 2y ago
I understand where you are coming from and that it should be assumed a vendor box can do anything even without SSH. Other teams don't think this way however. "Oh its just outbound HTTPS, no inbound connections, totally fine" is the school of thought I was up against. Adding to this the box was expected to send hundreds of GB of logs to do "big-data" against daily to monitor it's health, thus training firewalls, IDS and humans to expect this outbound data volume, which could with the flip of a bit start exfiltrating customer data. If I told you what this box was for you would know how insane it would be to send more than 500KB per day for that purpose. It seems I share something in common with Boeing "line stoppers". There are details I wish I could share as they would drive home how insane this entire debacle was. I expect to read about it in the news some day.
- Silhouette 2y ago
  This is always the problem with firewalls. If your adversary controls both ends of the connection then it's not outbound HTTPS traffic you're letting through. It's totally arbitrary two-way traffic that happens to be transmitted over a connection first established to port 443 on the remote host. The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it. In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.
- j-bos 2y ago
  Sounds like a situation needing a whistleblower.
  - BonoboIO 2y ago

BeefySwain 2y ago

You mention a couple times that you can't give more details. Can you say why? NDA?

LinuxBender 2y ago
Multiple NDA's among other things.

windexh8er 2y ago

> A firewall vendor claimed they could block anything inside that outbound HTTPS connection that was not HTTP but they could not. This is very easily bypassed leveraging cert pinning. Modern firewalling is all predicted on MitM approach, nobody has any secret sauce here. If they can't see inside the encryption they really can't do much. Very few customers have decryption configured correctly, or at all, at scale. Also an enterprise generally won't block connections that "aren't categorized" (URL blocklist) because it's too much work / headache Beyond that most good and bad actors have domains lying around that won't end up in blocked categories. NGFW today are NGDS (Next Gen Door Stops), they aren't effective beyond controlling their own users. And at that rate DNS is a much more cost effective control.

pixl97 2y ago
>Also an enterprise generally won't block connections that "aren't categorized" Depends where. I work with a lot of large enterprise and they absolutely do block everything. Anything leaving their data centers is proxied and allow listed by the proxy. If we tried to cert pin our application, it would immediately break in their environment and would not be allowed till it passed their policies.
- windexh8er 2y ago
  There are still many ways around this. A proxy is only as good as what the administrators have thought of as bypass. Things like domain fronting are still easily leveraged. And most organizations won't touch financial websites with a 10-foot pole because of the legal obligation of potentially decrypting PII. It's not impossible to get a domain classified as financial with a bit of work. The unfortunate reality is all it takes is one.