3 ms·

> Our basic philosophy when it comes to security is that we can trust our developers and that we can trust the private network within the cluster. As an infose

> Our basic philosophy when it comes to security is that we can trust our developers and that we can trust the private network within the cluster. As an infosec guy, I hate to say it but this is IMO very misguided. Insider attacks and external attacks are often indistinguishable because attackers are happy to steal developer credentials or infect their laptops with malware. Same with trusting the private network. That’s fine and dandy until attackers are in your network, and now they have free rein because you assumed you could keep the bad people outside the walls protecting your soft, squishy insides.

jonstewart 2y ago
One of the best things you can do is restrict your VPCs from accessing the internet willy-nilly outbound. When an attacker breaches you, this can keep them from downloading payloads and exfiltrating data.
- sanderjd 2y ago
  In the scenario presented, can't they just exfiltrate using the developer credentials / machine?
  - deleted 2y ago
    [deleted]
  - jonstewart 2y ago
    Let’s say there’s a log4j-type vuln and your app is affected. So an attacker can trigger an RCE in your app, which is running in, say, an EC2 instance in a VPC. A well-configured app server instance will have only necessary packages on it, and hopefully not much for dev tools. The instance will also run with certain privileges through IAM and then there won’t be creds on the instance for the attacker to steal. Typically an RCE like this runs a small script that will download and run a more useful piece of malware, like a webshell. If the webshell doesn’t download, the attacker probably is moving onto the next victim.
    - sanderjd 2y ago
      But the original comment wasn't about this attack vector... > attackers are happy to steal developer credentials or infect their laptops with malware I don't think any of what you said applies when an attacker has control of a developer machine that is allowed inside the network.
      - bigfatkitten 2y ago
        That is also a risk. Random developer machines being able to just connect to whatever they like inside prod is another poor architectural choice.
      - jonstewart 2y ago
        I was responding more to "Same with trusting the private network. That’s fine and dandy until attackers are in your network, and now they have free rein because you assumed you could keep the bad people outside the walls protecting your soft, squishy insides." Obviously this can apply to insiders in a typical corporate network, but it also applies to trust in a prod VPC environment.
- jiggawatts 2y ago
  You’ve just broken a hundred things that developers and ops staff need daily to block a theoretical vulnerability that is irrelevant unless you’re already severely breached. This kind of thinking is why secops often develops an adversarial relationship with other teams — the teams actually making money. I’ve seen this dynamic play out dozens of times and I’ve never seen it block an attack. I have seen it tank productivity and break production systems many times however. PS: The biggest impact denying outbound traffic has is to block Windows Update or the equivalent for other operating systems or applications. I’m working with a team right now that has to smuggle NPM modules in from their home PCs because they can’t run “npm audit fix” successfully on their isolated cloud PCs. Yes, for security they’re prevented from updating vulnerable packages unless they bend over backwards.
  - bigfatkitten 2y ago
    > I’ve seen this dynamic play out dozens of times and I’ve never seen it block an attack. I am a DFIR consultant, and I've been involved in 20 or 30 engagements over the last 15 years where proper egress controls would've stopped the adversary in their tracks.
    - jiggawatts 2y ago
      Any statement like that qualified with “proper” is a no true Scotsman fallacy. What do you consider proper egress blocking? No DNS? No ICMP? No access to any web proxy? No CDP or OCSP access? Strict domain-based filtering of all outbound traffic? What about cloud management endpoints? This can get to the point that it becomes nigh impossible to troubleshoot anything. Not even “ping” works! And troubleshoot you will have to, trust me. You’ll discover that root cert updates are out-of-band and not included in some other security patches. And you’ll discover that the 60s delay that’s impossible to pin down is a CRL validating timeout. You’ll discover that ICMP isn’t as optional as you thought. I’ve been that engineer, I’ve done this work, and I consider it a waste of time unless it is protecting at least a billion dollars worth of secrets. PS: practically 100% of exfiltrated data goes via established and approved channels such as OneDrive. I just had a customer send a cloud VM disk backup via SharePoint to a third party operating in another country. Oh, not to mention the telco that has outsourced core IT functions to both Chinese and Russian companies. No worries though! They’ve blocked me from using ping to fix their broken network.
bigfatkitten 2y ago
It's a mindset that keeps people like you and I employed in well-paying jobs.
apitman 2y ago
What's your opinion on EDR in general? I find it very distasteful from a privacy perspective, but obviously it could be beneficial at scale. I just wish there was a better middle ground.
- yodelshady 2y ago
  Not the OP but I was on that side - They do work. My best analogy is it's like working at TSA except there are three terrorist attacks per week. As far as privacy goes, by the same analogy, I can guarantee the operators don't care what porn you watch. Doing the job is more important. But still, treat your work machine as a work machine. It's not yours, it's a tool your company lent to you to work with. That said, on HN your workers are likely to be developers - that does take some more skill, and I'd advise asking a potential provider frank questions about their experience with the sector, as well as your risk tolerance. Devs do dodgy stuff all the time, and they usually know what they're doing, but when they don't you're going to have real fun proving you've remediated.
- tryauuum 2y ago
  EDR is not related to the topic but now I'm curious as well. Any good EDR for ubuntu server?

jonstewart 2y ago

One of the best things you can do is restrict your VPCs from accessing the internet willy-nilly outbound. When an attacker breaches you, this can keep them from downloading payloads and exfiltrating data.

sanderjd 2y ago
In the scenario presented, can't they just exfiltrate using the developer credentials / machine?
- deleted 2y ago
  [deleted]
- jonstewart 2y ago
  Let’s say there’s a log4j-type vuln and your app is affected. So an attacker can trigger an RCE in your app, which is running in, say, an EC2 instance in a VPC. A well-configured app server instance will have only necessary packages on it, and hopefully not much for dev tools. The instance will also run with certain privileges through IAM and then there won’t be creds on the instance for the attacker to steal. Typically an RCE like this runs a small script that will download and run a more useful piece of malware, like a webshell. If the webshell doesn’t download, the attacker probably is moving onto the next victim.
  - sanderjd 2y ago
    But the original comment wasn't about this attack vector... > attackers are happy to steal developer credentials or infect their laptops with malware I don't think any of what you said applies when an attacker has control of a developer machine that is allowed inside the network.
    - bigfatkitten 2y ago
      That is also a risk. Random developer machines being able to just connect to whatever they like inside prod is another poor architectural choice.
    - jonstewart 2y ago
      I was responding more to "Same with trusting the private network. That’s fine and dandy until attackers are in your network, and now they have free rein because you assumed you could keep the bad people outside the walls protecting your soft, squishy insides." Obviously this can apply to insiders in a typical corporate network, but it also applies to trust in a prod VPC environment.
jiggawatts 2y ago
You’ve just broken a hundred things that developers and ops staff need daily to block a theoretical vulnerability that is irrelevant unless you’re already severely breached. This kind of thinking is why secops often develops an adversarial relationship with other teams — the teams actually making money. I’ve seen this dynamic play out dozens of times and I’ve never seen it block an attack. I have seen it tank productivity and break production systems many times however. PS: The biggest impact denying outbound traffic has is to block Windows Update or the equivalent for other operating systems or applications. I’m working with a team right now that has to smuggle NPM modules in from their home PCs because they can’t run “npm audit fix” successfully on their isolated cloud PCs. Yes, for security they’re prevented from updating vulnerable packages unless they bend over backwards.
- bigfatkitten 2y ago
  > I’ve seen this dynamic play out dozens of times and I’ve never seen it block an attack. I am a DFIR consultant, and I've been involved in 20 or 30 engagements over the last 15 years where proper egress controls would've stopped the adversary in their tracks.
  - jiggawatts 2y ago
    Any statement like that qualified with “proper” is a no true Scotsman fallacy. What do you consider proper egress blocking? No DNS? No ICMP? No access to any web proxy? No CDP or OCSP access? Strict domain-based filtering of all outbound traffic? What about cloud management endpoints? This can get to the point that it becomes nigh impossible to troubleshoot anything. Not even “ping” works! And troubleshoot you will have to, trust me. You’ll discover that root cert updates are out-of-band and not included in some other security patches. And you’ll discover that the 60s delay that’s impossible to pin down is a CRL validating timeout. You’ll discover that ICMP isn’t as optional as you thought. I’ve been that engineer, I’ve done this work, and I consider it a waste of time unless it is protecting at least a billion dollars worth of secrets. PS: practically 100% of exfiltrated data goes via established and approved channels such as OneDrive. I just had a customer send a cloud VM disk backup via SharePoint to a third party operating in another country. Oh, not to mention the telco that has outsourced core IT functions to both Chinese and Russian companies. No worries though! They’ve blocked me from using ping to fix their broken network.

bigfatkitten 2y ago

It's a mindset that keeps people like you and I employed in well-paying jobs.

apitman 2y ago

What's your opinion on EDR in general? I find it very distasteful from a privacy perspective, but obviously it could be beneficial at scale. I just wish there was a better middle ground.

yodelshady 2y ago
Not the OP but I was on that side - They do work. My best analogy is it's like working at TSA except there are three terrorist attacks per week. As far as privacy goes, by the same analogy, I can guarantee the operators don't care what porn you watch. Doing the job is more important. But still, treat your work machine as a work machine. It's not yours, it's a tool your company lent to you to work with. That said, on HN your workers are likely to be developers - that does take some more skill, and I'd advise asking a potential provider frank questions about their experience with the sector, as well as your risk tolerance. Devs do dodgy stuff all the time, and they usually know what they're doing, but when they don't you're going to have real fun proving you've remediated.
tryauuum 2y ago
EDR is not related to the topic but now I'm curious as well. Any good EDR for ubuntu server?