5 ms·

Increasing Attacker Cost Using Immutable Infrastructure

110 points by diogomonicapt 10y ago

axelfontaine 10y ago
This is definitely one of the main reason we decided to this road for VMs as well at Boxfuse (https://boxfuse.com https://boxfuse.com): minimal images (5 MB for a full VM with a Go application) and immutable infrastructure all the way.
thinkmassive 10y ago
I know the author is pitching the immutable infrastructure side, which definitely has its merits. But the ability to so easily diff the image and running state of a container also opens some intriguing opportunities for honeypot automation. Are there any open source solutions that already take advantage of features like this? Or are those mostly kept secret for security and business reasons at this time?
- zokier 10y ago
  Intriguing indeed. Something like when a write happens, instead of just plain blocking it redirect the write to a honeypot environment and send out an alert. Sure, that won't fool a good attacker for a very long time, but would be interesting to capture various drive-by and skriptkiddie attacks.
oolongCat 10y ago
So I am confused, how does any of this increase attacker cost? Cant you do the same thing at the OS level already? Wouldn't making the dir read only do the same thing? If your data gets stolen and your website defaced what's the use of immutability? I mean you can always run a diff tool against the current code on the server with the code you have on your repo right?
- dgoldstein0 10y ago
  it's pretty hard to diff filesystems that aren't designed for it. Either you need to lock the whole filesystem somehow - e.g. by taking it offline - or you have to deal with the fact that other processes are reading/writing as you scan the filesystem, which is rather difficult to reason about. And it's not just about diffing your code with your repo - that only works if the attacker tried to attack your code. What about other running processes? New files on the system containing malicious code, outside of the paths you usually deploy code to? what about new, unexpected cron jobs? Overall, it could become a pretty complex job. A filesystem with some intrinsic snapshotting makes this a lot easier.
  - icebraining 10y ago
    You can also use lvm snapshots, which work with any filesystem.
  - Senji 10y ago
    COW by default file systems which provide a "snapshot at time x" can help with this. NTFS can do it with shadowcopy.
- viraptor 10y ago
  > how does any of this increase attacker cost? Because it forces the attacker to write a specific payload for your service. Standard, reused "drop shell.php and register IP" will not work anymore. And realistically if the target of the attack was a WordPress installation, it will likely be a trivial, automated script. > Cant you do the same thing at the OS level already? Yes, you can. Even better, split execution privileges from file privileges, then make it read only, then put a grsec/apparmor/selinux profile on the service. It's not docker specific, but docker does make read only service a little bit easier. > Wouldn't making the dir read only do the same thing? Yeah, but who would do that old school thing. Docker security! :-(
xorcist 10y ago
It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage. The article tries to pitch read only Docker images some kind of solution, but running your applications read only (and what other permissions you grant your application) has nothing to do with Docker images. Using file system and process namespaces for application isolation is a good idea. But lately it seems to be getting more popular to drop untrusted applications in containers (possibly even under outside control and full of who-knows-what) as if that somehow solves "security". The thing is, you still need to be able to reason about what permissions your respective application requires, there's no getting away from that.
- Senji 10y ago
  Should probably just tiger-tree-hash your containers. If they deviate you give them a copy of your database, but it's a "virtual" shadow copy. No writes to it reflect on the real database.
  - toomuchtodo 10y ago
    If attackers get your database, you lose. Containers don't fix application vulnerability mitigation and leaking data.
- drieddust 10y ago
  > It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage. I would also add that attackers are actually after the data. Exploiting application vulnerabilities is just a mean to that end so bringing back exploitable application from the previous image is a BAD idea. Having said that previous image can be a good starting point to patching up the vulnerabilities and bringing the application online.
  - 10y ago
JoachimSchipper 10y ago
Docker is not (designed to be) a security technology. Yes, rolling back servers/VMs/containers to their previous state is a good capability to have (although most of us just use backups for that!), but assuming that an attacker cannot break out of a container is, at least, optimistic.
- wepple 10y ago
  > assuming that an attacker cannot break out of a container is, at least, optimistic. Agreed. However as long as you don't look at it as your primary line of defense, it increases the cost to an attacker. And that's currently the best we can ever do.
  - JoachimSchipper 10y ago
    There are arguments - which are at least plausible - that the same effort is better spent on VM / Solaris Zones / FreeBSD jails / traditional chroot / SELinux / just using dedicated hardware for everything / ...; several of these have the advantage of not encouraging people to throw up a compartment (of some kind) and never update it again, which is certainly something that happens with Docker.
sublimino 10y ago
Of note is that an immutable/noexec filesystem doesn't prevent code being downloaded to an environment var/typed out and run - tools like https://github.com/SafeBreach-Labs/pwndsh https://github.com/SafeBreach-Labs/pwndsh just pipe source to an interpreter (in that case BASH, which generally isn't installed in smaller base images). Reducing the attack surface is important, but if a running container is compromised it's imperative a post-mortem is performed immediately - and the issue remediated - to prevent re-exploitation.
- justincormack 10y ago
  potentially you do not need any interpreters available at all, which certainly increases attack difficulty.
runeks 10y ago
> Until we fix this RCE vulnerability, the attacker will > still be able to execute code on our host [...] With Docker, it seems to me like we're moving closer and closer to the server being an executable of its own, but with the necessary Linux kernel bits compiled in such that it can execute on (virtualized) hardware. I'm wondering how far we can take this. The ability to execute code on the host is there because that's what Linux does, but what if we removed this interface, and replaced it with an interface that just accepts one or more ELF binaries at compile-time? Then these would become the only Linux executables that this kernel can execute. As far as I can see, we could do the same to system calls: if an executable can enumerate all the system calls it needs, we can compile a kernel that will accept only these system calls, which should be a small subset of all available Linux syscalls.
- stable-point 10y ago
  This is called a Unikernel: https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel > Unikernels are specialised, single address space machine images constructed by using library operating systems. A developer selects, from a modular stack, the minimal set of libraries which correspond to the OS constructs required for their application to run. These libraries are then compiled with the application and configuration code to build sealed, fixed-purpose images (unikernels) which run directly on a hypervisor or hardware without an intervening OS such as Linux or Windows.
  - runeks 10y ago
    That is correct. I guess what I wanted to point out is that it would be cool if Linux could become this - a library that you can compile into your program, rather than a program in which you run your programs (an OS).
    - paulddraper 10y ago
      Linux is purposefully and by design monolithic. However, you could base your unikernal capabilities on the same interface (like POSIX). Current unikernals: http://unikernel.org/projects/ http://unikernel.org/projects/
Animats 10y ago
Real immutability would be good in more applications, as when you load a program into the memory of an embedded processor and then blow the programming fuse so it can never be changed. Hardware where parts of memory can be made read-only after boot and can't be written again without a full processor reboot would be useful.
bureado 10y ago
I remember when mounting /tmp noexec and chmod g-w on htdocs was enough.

110 points by diogomonicapt 10y ago

This is definitely one of the main reason we decided to this road for VMs as well at Boxfuse (https://boxfuse.com https://boxfuse.com): minimal images (5 MB for a full VM with a Go application) and immutable infrastructure all the way.

thinkmassive 10y ago

I know the author is pitching the immutable infrastructure side, which definitely has its merits. But the ability to so easily diff the image and running state of a container also opens some intriguing opportunities for honeypot automation. Are there any open source solutions that already take advantage of features like this? Or are those mostly kept secret for security and business reasons at this time?

zokier 10y ago
Intriguing indeed. Something like when a write happens, instead of just plain blocking it redirect the write to a honeypot environment and send out an alert. Sure, that won't fool a good attacker for a very long time, but would be interesting to capture various drive-by and skriptkiddie attacks.

oolongCat 10y ago

So I am confused, how does any of this increase attacker cost? Cant you do the same thing at the OS level already? Wouldn't making the dir read only do the same thing? If your data gets stolen and your website defaced what's the use of immutability? I mean you can always run a diff tool against the current code on the server with the code you have on your repo right?

dgoldstein0 10y ago
it's pretty hard to diff filesystems that aren't designed for it. Either you need to lock the whole filesystem somehow - e.g. by taking it offline - or you have to deal with the fact that other processes are reading/writing as you scan the filesystem, which is rather difficult to reason about. And it's not just about diffing your code with your repo - that only works if the attacker tried to attack your code. What about other running processes? New files on the system containing malicious code, outside of the paths you usually deploy code to? what about new, unexpected cron jobs? Overall, it could become a pretty complex job. A filesystem with some intrinsic snapshotting makes this a lot easier.
- icebraining 10y ago
  You can also use lvm snapshots, which work with any filesystem.
- Senji 10y ago
  COW by default file systems which provide a "snapshot at time x" can help with this. NTFS can do it with shadowcopy.
viraptor 10y ago
> how does any of this increase attacker cost? Because it forces the attacker to write a specific payload for your service. Standard, reused "drop shell.php and register IP" will not work anymore. And realistically if the target of the attack was a WordPress installation, it will likely be a trivial, automated script. > Cant you do the same thing at the OS level already? Yes, you can. Even better, split execution privileges from file privileges, then make it read only, then put a grsec/apparmor/selinux profile on the service. It's not docker specific, but docker does make read only service a little bit easier. > Wouldn't making the dir read only do the same thing? Yeah, but who would do that old school thing. Docker security! :-(

xorcist 10y ago

It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage. The article tries to pitch read only Docker images some kind of solution, but running your applications read only (and what other permissions you grant your application) has nothing to do with Docker images. Using file system and process namespaces for application isolation is a good idea. But lately it seems to be getting more popular to drop untrusted applications in containers (possibly even under outside control and full of who-knows-what) as if that somehow solves "security". The thing is, you still need to be able to reason about what permissions your respective application requires, there's no getting away from that.

Senji 10y ago
Should probably just tiger-tree-hash your containers. If they deviate you give them a copy of your database, but it's a "virtual" shadow copy. No writes to it reflect on the real database.
- toomuchtodo 10y ago
  If attackers get your database, you lose. Containers don't fix application vulnerability mitigation and leaking data.
drieddust 10y ago
> It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage. I would also add that attackers are actually after the data. Exploiting application vulnerabilities is just a mean to that end so bringing back exploitable application from the previous image is a BAD idea. Having said that previous image can be a good starting point to patching up the vulnerabilities and bringing the application online.
- 10y ago

JoachimSchipper 10y ago

Docker is not (designed to be) a security technology. Yes, rolling back servers/VMs/containers to their previous state is a good capability to have (although most of us just use backups for that!), but assuming that an attacker cannot break out of a container is, at least, optimistic.

wepple 10y ago
> assuming that an attacker cannot break out of a container is, at least, optimistic. Agreed. However as long as you don't look at it as your primary line of defense, it increases the cost to an attacker. And that's currently the best we can ever do.
- JoachimSchipper 10y ago
  There are arguments - which are at least plausible - that the same effort is better spent on VM / Solaris Zones / FreeBSD jails / traditional chroot / SELinux / just using dedicated hardware for everything / ...; several of these have the advantage of not encouraging people to throw up a compartment (of some kind) and never update it again, which is certainly something that happens with Docker.

sublimino 10y ago

Of note is that an immutable/noexec filesystem doesn't prevent code being downloaded to an environment var/typed out and run - tools like https://github.com/SafeBreach-Labs/pwndsh https://github.com/SafeBreach-Labs/pwndsh just pipe source to an interpreter (in that case BASH, which generally isn't installed in smaller base images). Reducing the attack surface is important, but if a running container is compromised it's imperative a post-mortem is performed immediately - and the issue remediated - to prevent re-exploitation.

justincormack 10y ago
potentially you do not need any interpreters available at all, which certainly increases attack difficulty.

runeks 10y ago

> Until we fix this RCE vulnerability, the attacker will > still be able to execute code on our host [...] With Docker, it seems to me like we're moving closer and closer to the server being an executable of its own, but with the necessary Linux kernel bits compiled in such that it can execute on (virtualized) hardware. I'm wondering how far we can take this. The ability to execute code on the host is there because that's what Linux does, but what if we removed this interface, and replaced it with an interface that just accepts one or more ELF binaries at compile-time? Then these would become the only Linux executables that this kernel can execute. As far as I can see, we could do the same to system calls: if an executable can enumerate all the system calls it needs, we can compile a kernel that will accept only these system calls, which should be a small subset of all available Linux syscalls.

stable-point 10y ago
This is called a Unikernel: https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel > Unikernels are specialised, single address space machine images constructed by using library operating systems. A developer selects, from a modular stack, the minimal set of libraries which correspond to the OS constructs required for their application to run. These libraries are then compiled with the application and configuration code to build sealed, fixed-purpose images (unikernels) which run directly on a hypervisor or hardware without an intervening OS such as Linux or Windows.
- runeks 10y ago
  That is correct. I guess what I wanted to point out is that it would be cool if Linux could become this - a library that you can compile into your program, rather than a program in which you run your programs (an OS).
  - paulddraper 10y ago
    Linux is purposefully and by design monolithic. However, you could base your unikernal capabilities on the same interface (like POSIX). Current unikernals: http://unikernel.org/projects/ http://unikernel.org/projects/

Animats 10y ago

Real immutability would be good in more applications, as when you load a program into the memory of an embedded processor and then blow the programming fuse so it can never be changed. Hardware where parts of memory can be made read-only after boot and can't be written again without a full processor reboot would be useful.

bureado 10y ago

I remember when mounting /tmp noexec and chmod g-w on htdocs was enough.