4 ms·

I understand where you are coming from and that it should be assumed a vendor box can do anything even without SSH. Other teams don't think this way however.

I understand where you are coming from and that it should be assumed a vendor box can do anything even without SSH. Other teams don't think this way however. "Oh its just outbound HTTPS, no inbound connections, totally fine" is the school of thought I was up against. Adding to this the box was expected to send hundreds of GB of logs to do "big-data" against daily to monitor it's health, thus training firewalls, IDS and humans to expect this outbound data volume, which could with the flip of a bit start exfiltrating customer data. If I told you what this box was for you would know how insane it would be to send more than 500KB per day for that purpose. It seems I share something in common with Boeing "line stoppers". There are details I wish I could share as they would drive home how insane this entire debacle was. I expect to read about it in the news some day.

Silhouette 2y ago
This is always the problem with firewalls. If your adversary controls both ends of the connection then it's not outbound HTTPS traffic you're letting through. It's totally arbitrary two-way traffic that happens to be transmitted over a connection first established to port 443 on the remote host. The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it. In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.
j-bos 2y ago
Sounds like a situation needing a whistleblower.
- BonoboIO 2y ago
  Hopefully not ending like the Boeing whistleblowers
Bluecobra 2y ago
This is a good point. I would expect that a device like a SAN is just sending telemetry/logs/diagnostics back to the mothership for support purposes. Having a a persistent tunnel kinda of sucks and I much prefer something like shell access being done over a remote support/screen sharing app so I can see what they are doing. Previous security fiascos like the Solarwinds hack come to mind and an attacker could have a foothold inside a trusted/internal network.
- nine_k 2y ago
  To be properly paranoid, I would allow the device to send telemetry and diagnostics, but only through my proxy. The outbound stream can be as encrypted as they want, but I will demand the ability to decode the answer, and decide whether I let it come back to the box. I wonder how many vendors would agree to offer this, and how much more would t then cost. (If you update software from the vendor's resource, all bets are off, because you just run their software which can do anything your security measures would not prevent it from doing. You have to very seriously trust the vendor of your OS, if you may be a high-value target.)
  - Spooky23 2y ago
    If you’re big enough, they will. One company fought it, so I stopped paying them and they found Jesus.
    - marcosdumay 2y ago
      That is more of a case of that one vendor not having diverse enough customers. Not of your company being big.
      - Spooky23 2y ago
        Nah. I’ve seen similar stunts pulled off with companies like Microsoft. Sales teams who believe a full funnel is in front of them are capable of incredible feats. You need to have the aircover and willingness to scorch the earth.
commandlinefan 2y ago
But it's their device, you're installing it in your network - if they wanted to do something malicious, they would. If you shut down the tunneling method detailed in the article, they could just add an endpoint like: GET /latest-command that resolved to a shell script to be run periodically. > start exfiltrating customer data If it's data that they're supposed to have access to, they're already doing that. If it's data they're not supposed to have access to, the correct fix here is to DMZ the box they're installed on, not to try to (hopelessly) limit their outbound connectivity.

Silhouette 2y ago

This is always the problem with firewalls. If your adversary controls both ends of the connection then it's not outbound HTTPS traffic you're letting through. It's totally arbitrary two-way traffic that happens to be transmitted over a connection first established to port 443 on the remote host. The only useful technical defence against this kind of deception today is deep packet inspection and a policy of blocking everything by default and only permitting through packets you can actively approve. But that becomes very expensive very quickly and there are practical limits on how far you can go. Ultimately if your adversary is willing to engage covertly in the kind of hacks mentioned in the article then they're probably also willing to engage in steganography to get past whatever DPI rules you can afford to run. Then you're back to square one and either you trust their device or you sandbox it. In reality a more effective defence is probably the one involving contracts with severe penalties for this kind of behaviour and liability for any consequential losses.

j-bos 2y ago

Sounds like a situation needing a whistleblower.

BonoboIO 2y ago
Hopefully not ending like the Boeing whistleblowers

Bluecobra 2y ago

This is a good point. I would expect that a device like a SAN is just sending telemetry/logs/diagnostics back to the mothership for support purposes. Having a a persistent tunnel kinda of sucks and I much prefer something like shell access being done over a remote support/screen sharing app so I can see what they are doing. Previous security fiascos like the Solarwinds hack come to mind and an attacker could have a foothold inside a trusted/internal network.

nine_k 2y ago
To be properly paranoid, I would allow the device to send telemetry and diagnostics, but only through my proxy. The outbound stream can be as encrypted as they want, but I will demand the ability to decode the answer, and decide whether I let it come back to the box. I wonder how many vendors would agree to offer this, and how much more would t then cost. (If you update software from the vendor's resource, all bets are off, because you just run their software which can do anything your security measures would not prevent it from doing. You have to very seriously trust the vendor of your OS, if you may be a high-value target.)
- Spooky23 2y ago
  If you’re big enough, they will. One company fought it, so I stopped paying them and they found Jesus.
  - marcosdumay 2y ago
    That is more of a case of that one vendor not having diverse enough customers. Not of your company being big.
    - Spooky23 2y ago
      Nah. I’ve seen similar stunts pulled off with companies like Microsoft. Sales teams who believe a full funnel is in front of them are capable of incredible feats. You need to have the aircover and willingness to scorch the earth.

commandlinefan 2y ago

But it's their device, you're installing it in your network - if they wanted to do something malicious, they would. If you shut down the tunneling method detailed in the article, they could just add an endpoint like: GET /latest-command that resolved to a shell script to be run periodically. > start exfiltrating customer data If it's data that they're supposed to have access to, they're already doing that. If it's data they're not supposed to have access to, the correct fix here is to DMZ the box they're installed on, not to try to (hopelessly) limit their outbound connectivity.