5 ms·

> In other words, it’s less a question of how to stop hackers from breaking in than how to best survive the inevitable damage. There doesn't seem to be convent

> In other words, it’s less a question of how to stop hackers from breaking in than how to best survive the inevitable damage. There doesn't seem to be conventional wisdom about how to build systems that are easy to restore. How do you optimize for recovery after an attack? How do you ensure that you've eliminated all the backdoors? My guess is a combination of "continuous restoration", version controlled code, and a complete separation of code from data. I want to read books about this but they don't seem to exist.

t-shaped 6y ago
If there's a book that you want to read, but it hasn't been written yet, then you must write it. - Toni Morrison
jeffbee 6y ago
Just having a decent and reasonable way to nuke and pave machines goes a long way. Most organizations don't have a good way to shoot a machine in the face and have it back up and serving in 2 minutes. Most organizations are absolutely married to "stateful services" like SQL databases with local storage, that are hard to kill, hard to restore, and give attackers a place to hang out. If you can take all your hosts down and bring them all back up quickly, that gives you at least one tool for disrupting the attackers.
- jonpurdy 6y ago
  I was thinking about this recently. Tight, centralized control over servers, employees computers, and devices is hard to set up, hard to manage, and a huge surface where misconfigurations can allow attackers to jump right in. Decentralization is the key. Microservices, or segregated services, stateless (as much as possible), and perhaps even partitioning groups of users into totally separated instances. One group gets attacked and service only goes out for the 1000 users in the group. Of course, infrastructure costs would go up but maybe not that much (since you need less resources for 1000 users than 100,000). This is relatively easy to build from scratch these days thanks to various IaaS providers and DevOps tools (obviously hard for established companies with legacy tech). Then, keep employee computers totally separate from production servers. Let employees back them up themselves, especially since so much can just be stored in the cloud (I'll get there next). Forget about VPNs where everyone can talk to everyone else. Don't try to save money by hosting your own Jira and Bitbucket instances. Pay extra for Atlassian to host for you, then let them deal with security (actually it's cheaper from what I remember). Companies already pay for Office 365 and don't self-host that. Don't host your own email servers. Just focus on the core of what the company needs to do and that's it. This way you spread your attack surface across a whole bunch of services that are better than you at security, and you get the benefit of not having to deal with other issues. If Atlassian or Cloudflare goes down, no big deal; they'll fix it. And all your other stuff still works.
  - square_dog 6y ago
    Historically, centralization in the computer world grew from the glass house in which the one computer a company or institution could afford to buy or lease was installed -- and programmed by the tech gurus colocated there. The centralization and batch processing constraints of that era are now obsolete, having been replaced by inexpensive distributable computing, data storage and network building blocks. I suspect that many organizations haven't fully embraced the far more reliable, dependable and scalable approach to delivering services is related more to managerial issues ("tried and proven") than technical creativity and ability. The "C-Team" needs to be sold on the benefits that are obvious to the tech gurus of today, in language corporate leadership can understand and motivate their commitment.
- em-bee 6y ago
  which company was it that randomly shot down machines in production to make sure the system would be resilient enough?
  - m12k 6y ago
    Sounds like the Netflix chaos engineering team
  - toxik 6y ago
    Wrench monkey was an Amazon idea I think
  - deleted 6y ago
    [deleted]
- coolgeek 6y ago
  This comment works equally well as nerd metal lyrics.
russdill 6y ago
One of the things I loved when working at a large corp with military contracts. Anything fishy from any PC and the PC was gone. No notice to the employee, no waiting, no nothing. It was removed from the network, wiped, and reimaged as quickly as possible.
GordonS 6y ago
There are often 2 threats from a randsomware attack - what you describe mitigates against data loss, but not against data being posted online by an attacker.
- rectang 6y ago
  I disagree. Restoring quickly to a known-good state is a crucial aspect of shutting down an attack after the intrusion is detected. If you can't get the attacker out of the system, they will have more opportunity to exfiltrate data.
  - GordonS 6y ago
    I'm not disagreeing with having the ability to restore your data - plainly that's vital to business continuity. My point is that it's not the only consideration.

t-shaped 6y ago

If there's a book that you want to read, but it hasn't been written yet, then you must write it. - Toni Morrison

jeffbee 6y ago

Just having a decent and reasonable way to nuke and pave machines goes a long way. Most organizations don't have a good way to shoot a machine in the face and have it back up and serving in 2 minutes. Most organizations are absolutely married to "stateful services" like SQL databases with local storage, that are hard to kill, hard to restore, and give attackers a place to hang out. If you can take all your hosts down and bring them all back up quickly, that gives you at least one tool for disrupting the attackers.

jonpurdy 6y ago
I was thinking about this recently. Tight, centralized control over servers, employees computers, and devices is hard to set up, hard to manage, and a huge surface where misconfigurations can allow attackers to jump right in. Decentralization is the key. Microservices, or segregated services, stateless (as much as possible), and perhaps even partitioning groups of users into totally separated instances. One group gets attacked and service only goes out for the 1000 users in the group. Of course, infrastructure costs would go up but maybe not that much (since you need less resources for 1000 users than 100,000). This is relatively easy to build from scratch these days thanks to various IaaS providers and DevOps tools (obviously hard for established companies with legacy tech). Then, keep employee computers totally separate from production servers. Let employees back them up themselves, especially since so much can just be stored in the cloud (I'll get there next). Forget about VPNs where everyone can talk to everyone else. Don't try to save money by hosting your own Jira and Bitbucket instances. Pay extra for Atlassian to host for you, then let them deal with security (actually it's cheaper from what I remember). Companies already pay for Office 365 and don't self-host that. Don't host your own email servers. Just focus on the core of what the company needs to do and that's it. This way you spread your attack surface across a whole bunch of services that are better than you at security, and you get the benefit of not having to deal with other issues. If Atlassian or Cloudflare goes down, no big deal; they'll fix it. And all your other stuff still works.
- square_dog 6y ago
  Historically, centralization in the computer world grew from the glass house in which the one computer a company or institution could afford to buy or lease was installed -- and programmed by the tech gurus colocated there. The centralization and batch processing constraints of that era are now obsolete, having been replaced by inexpensive distributable computing, data storage and network building blocks. I suspect that many organizations haven't fully embraced the far more reliable, dependable and scalable approach to delivering services is related more to managerial issues ("tried and proven") than technical creativity and ability. The "C-Team" needs to be sold on the benefits that are obvious to the tech gurus of today, in language corporate leadership can understand and motivate their commitment.
em-bee 6y ago
which company was it that randomly shot down machines in production to make sure the system would be resilient enough?
- m12k 6y ago
  Sounds like the Netflix chaos engineering team
- toxik 6y ago
  Wrench monkey was an Amazon idea I think
- deleted 6y ago
  [deleted]
coolgeek 6y ago
This comment works equally well as nerd metal lyrics.

russdill 6y ago

One of the things I loved when working at a large corp with military contracts. Anything fishy from any PC and the PC was gone. No notice to the employee, no waiting, no nothing. It was removed from the network, wiped, and reimaged as quickly as possible.

GordonS 6y ago

There are often 2 threats from a randsomware attack - what you describe mitigates against data loss, but not against data being posted online by an attacker.

rectang 6y ago
I disagree. Restoring quickly to a known-good state is a crucial aspect of shutting down an attack after the intrusion is detected. If you can't get the attacker out of the system, they will have more opportunity to exfiltrate data.
- GordonS 6y ago
  I'm not disagreeing with having the ability to restore your data - plainly that's vital to business continuity. My point is that it's not the only consideration.