4 ms·

Developing in container environments with limited access might help, but I think there's a performance hit for heavy processing/ML training unless you use privi

by ftufek 4y ago

Developing in container environments with limited access might help, but I think there's a performance hit for heavy processing/ML training unless you use privileged mode which kinda defeats the purpose.

p-e-w 4y ago
Also, the container itself usually contains (or has access to) valuable secrets, such as keys for staging servers and, of course, the source code.
zvolsky 4y ago
What I do is that I do all dev work in a VM, usually dedicated to a group of related projects. There are no passwords stored in the VM and its ssh key has read-only access to repositories. The VM has write access to forks and code is merged using pull requests just like third party contributions. This setup has been working well for me, and I even created a tool to automate setting up these dev VMs which I hope to make publicly available at some point. I don't do any heavy computations except for running test suites of various kinds, and these seem to perform the same on raw hardware as they do in the Kernel-based VMs.
tkinom 4y ago
Love to see a container environment that can monitor Monitor and log all outgoing network connection requests.... Monitor and log all critical file/directory access such as /etc/* With such container, we can catch the compromised supply-chain attach easily, right? Does anyone know such container exist?
- munchbunny 4y ago
  Only using privileged containers, or else you don’t have visibility into signal from other containers. But, say you had such a container, there’s an important distinction between “you captured a log showing the smoking gun evidence of the supply chain attack”, and “you successfully picked that log out of all of the log data you generated and classified it with high confidence as an attack”. Speaking from experience, the second problem is the hard problem for a multitude of reasons. So while you would have the data, you’d probably have trouble getting good precision/recall on when to actually sound the alarms vs. when it’s some SRE who needed to troubleshoot some network connectivity issues.
  - yjftsjthsd-h 4y ago
    > Only using privileged containers, or else you don’t have visibility into signal from other containers. The suspect application doesn't need the privileges, so I'm not sure how much of a problem that is? > there’s an important distinction between “you captured a log showing the smoking gun evidence of the supply chain attack”, and “you successfully picked that log out of all of the log data you generated and classified it with high confidence as an attack”. Assuming that you're talking about the signal:noise problem, that's hard in the general case but I feel like you could easily pick off really obvious cases like trying to access private SSH/GPG keys and still get a lot of value.
    - munchbunny 4y ago
      > Assuming that you're talking about the signal:noise problem, that's hard in the general case but I feel like you could easily pick off really obvious cases like trying to access private SSH/GPG keys and still get a lot of value. Probably. I’d agree that it’s worth trying at the very least. I’ve run into enough “should be easy” cases that turn out to be not that easy that my default is to get the data and see if the hypothesis really pans out.
- ashishbijlani 4y ago
  I’ve created Packj sandbox [1] for “safe installation” of PyPI/NPM/Rubygems packages 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj It DOES NOT require a VM/Container; uses strace. It shows you a preview of file system changes that installation will make and can also block arbitrary network communication during installation (uses an allow-list).
  - remram 4y ago
    strace uses ptrace, which is not safe for security use because of race conditions. Linux Security Modules should be used. https://stackoverflow.com/a/4421762/711380 https://stackoverflow.com/a/4421762/711380
    - ashishbijlani 4y ago
      Thanks for highlighting this! While PTRACE introduces TOCTTOU vulnerabilities, Packj sandboxes fixes that by using read-only args for ptrace. You can find my PhD work [1] on this relevant. 1. https://lwn.net/Articles/803890/ https://lwn.net/Articles/803890/
- varunsharma07 4y ago
  If CI/ CD pipeline uses GitHub Actions, you can monitor and even block outbound network calls at the DNS and network level using Harden Runner (https://github.com/step-security/harden-runner https://github.com/step-security/harden-runner). It can also detect overwrite of files in the working directory. Harden Runner would have caught this dependency confusion and similar attacks due to a call to the attacker endpoint.
nextos 4y ago
Jails / sandboxes such as bwrap could be enough in this case by denying access to e.g. HOME files not explicitly whitelisted. Also a Little Snitch-like host-based firewall, which would request explicit permission to connect.
rogers18445 4y ago
This is the way things should be done by any competent developer. Generally a VM jail is preferable - firecracker or cloud-hypervisor(virtiofs & gpu passthrough) recommended. A proper namespace jail (eg. bwrap) is sufficient for 99.9% of cases. To break out of a properly configured namespace jail you would need to sacrifice a 0day.
- codedokode 4y ago
  It looks like totally overengineered solution to me. The CPU already has a protected mode which doesn't allow the program to access any files directly. Why do you need to run a VM which by the way is run from the kernel (privileged mode) in Linux? Why cannot you run untrusted programs in protected mode?
  - rogers18445 4y ago
    With containers the entire surface area of the kernel is available to attack (syscalls). With a VM the surface is restricted to the VMM and KVM. This is an oversimplification, there may be other protocols that are passed through or utilized, they would add to the surface.

by ftufek 4y ago

p-e-w 4y ago

Also, the container itself usually contains (or has access to) valuable secrets, such as keys for staging servers and, of course, the source code.

zvolsky 4y ago

What I do is that I do all dev work in a VM, usually dedicated to a group of related projects. There are no passwords stored in the VM and its ssh key has read-only access to repositories. The VM has write access to forks and code is merged using pull requests just like third party contributions. This setup has been working well for me, and I even created a tool to automate setting up these dev VMs which I hope to make publicly available at some point. I don't do any heavy computations except for running test suites of various kinds, and these seem to perform the same on raw hardware as they do in the Kernel-based VMs.

tkinom 4y ago

Love to see a container environment that can monitor Monitor and log all outgoing network connection requests.... Monitor and log all critical file/directory access such as /etc/* With such container, we can catch the compromised supply-chain attach easily, right? Does anyone know such container exist?

munchbunny 4y ago
Only using privileged containers, or else you don’t have visibility into signal from other containers. But, say you had such a container, there’s an important distinction between “you captured a log showing the smoking gun evidence of the supply chain attack”, and “you successfully picked that log out of all of the log data you generated and classified it with high confidence as an attack”. Speaking from experience, the second problem is the hard problem for a multitude of reasons. So while you would have the data, you’d probably have trouble getting good precision/recall on when to actually sound the alarms vs. when it’s some SRE who needed to troubleshoot some network connectivity issues.
- yjftsjthsd-h 4y ago
  > Only using privileged containers, or else you don’t have visibility into signal from other containers. The suspect application doesn't need the privileges, so I'm not sure how much of a problem that is? > there’s an important distinction between “you captured a log showing the smoking gun evidence of the supply chain attack”, and “you successfully picked that log out of all of the log data you generated and classified it with high confidence as an attack”. Assuming that you're talking about the signal:noise problem, that's hard in the general case but I feel like you could easily pick off really obvious cases like trying to access private SSH/GPG keys and still get a lot of value.
  - munchbunny 4y ago
    > Assuming that you're talking about the signal:noise problem, that's hard in the general case but I feel like you could easily pick off really obvious cases like trying to access private SSH/GPG keys and still get a lot of value. Probably. I’d agree that it’s worth trying at the very least. I’ve run into enough “should be easy” cases that turn out to be not that easy that my default is to get the data and see if the hypothesis really pans out.
ashishbijlani 4y ago
I’ve created Packj sandbox [1] for “safe installation” of PyPI/NPM/Rubygems packages 1. https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj It DOES NOT require a VM/Container; uses strace. It shows you a preview of file system changes that installation will make and can also block arbitrary network communication during installation (uses an allow-list).
- remram 4y ago
  strace uses ptrace, which is not safe for security use because of race conditions. Linux Security Modules should be used. https://stackoverflow.com/a/4421762/711380 https://stackoverflow.com/a/4421762/711380
  - ashishbijlani 4y ago
    Thanks for highlighting this! While PTRACE introduces TOCTTOU vulnerabilities, Packj sandboxes fixes that by using read-only args for ptrace. You can find my PhD work [1] on this relevant. 1. https://lwn.net/Articles/803890/ https://lwn.net/Articles/803890/
varunsharma07 4y ago
If CI/ CD pipeline uses GitHub Actions, you can monitor and even block outbound network calls at the DNS and network level using Harden Runner (https://github.com/step-security/harden-runner https://github.com/step-security/harden-runner). It can also detect overwrite of files in the working directory. Harden Runner would have caught this dependency confusion and similar attacks due to a call to the attacker endpoint.

nextos 4y ago

Jails / sandboxes such as bwrap could be enough in this case by denying access to e.g. HOME files not explicitly whitelisted. Also a Little Snitch-like host-based firewall, which would request explicit permission to connect.

rogers18445 4y ago

This is the way things should be done by any competent developer. Generally a VM jail is preferable - firecracker or cloud-hypervisor(virtiofs & gpu passthrough) recommended. A proper namespace jail (eg. bwrap) is sufficient for 99.9% of cases. To break out of a properly configured namespace jail you would need to sacrifice a 0day.

codedokode 4y ago
It looks like totally overengineered solution to me. The CPU already has a protected mode which doesn't allow the program to access any files directly. Why do you need to run a VM which by the way is run from the kernel (privileged mode) in Linux? Why cannot you run untrusted programs in protected mode?
- rogers18445 4y ago
  With containers the entire surface area of the kernel is available to attack (syscalls). With a VM the surface is restricted to the VMM and KVM. This is an oversimplification, there may be other protocols that are passed through or utilized, they would add to the surface.