6 ms·

We’ve been using Sysbox (https://github.com/nestybox/sysbox https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker wit

by lox 5y ago

We’ve been using Sysbox (https://github.com/nestybox/sysbox https://github.com/nestybox/sysbox) for our Buildkite based CI/CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM/STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.

xmodem 5y ago
Could you elaborate a bit more how you get the containers into their own IAM roles?
- lox 5y ago
  Yup, we have a sidecar process/container that runs for each job and assumes an AWS IAM Role for that specific pipeline (with constraints like whether it’s an approved PR as well). The credentials are provided to the job container via a volume mount. This allows us to have shared agents with very granular roles per-pipeline and job.
- nijave 5y ago
  Not sure if this applies to the parent, but one way this Buildkite Queues map pipelines to agents. Agents can be assigned IAM roles. If you want a certain build to run as an IAM role, you give it a queue where the agents have that role. For AWS, Buildkite has as a Cloud Formation stack that sets up auto scaling groups and some other resources for your agents to run.
  - xmodem 5y ago
    Most CI systems will have some way of assigning builds to groups of agents. But it would in some cases be useful to grant different privileges to different containers running on the same agent, which is what I understood OP to have.
- orf 5y ago
  AWS has IAM service accounts for containers. Comes for free with EKS, not sure how you’d do it without EKS. Basically it adds a signed web identity file into the container which can be used to assume roles.
  - captn3m0 5y ago
    Ref: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f...
  - otterley 5y ago
    Amazon ECS also offers task roles which do the same thing: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
- selecsosi 5y ago
  For ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
lima 5y ago
Never head of Sysbox before. At a first glance, the comparison table in their GitHub repo and on their website[1] has a number of inaccuracies which makes me question the quality of their engineering: — They claim that their solution has the same isolation level ("4 stars") than gVisor, unlike "standard containers", which are "2 stars" only (with Firecracker and Kubevirt being "5 stars). This is very wrong - as far as I can tell, they use regular Linux namespaces with some light eBPF-based filesystem emulation, while the vast majority of syscalls is still handled by the host kernel. Sorry, but this is still "2 stars" and far away from the isolation guarantees provided by gVisor (fully emulating the kernel in userspace, which is at the same level or even better than Firecracker) and nowhere close to a VM. — Somehow, regular VMs (Kubevirt) get a "speed" rating of only "2 stars" - worse than gVisor ("3 stars") and Firecracker ("4 stars"), even though they both rely on virtually the same virtualization technology. If anything, gVisor is the slowest but most efficient solution while QEMU maintains some performance advantage over Firecracker[2]. These are basically random scores, it's not a good first impression–if you do a detailed comparison like that, at least do a proper evaluation before giving your own product the best score! — They claim that "standard containers" cannot run a full OS. This isn't true - while it's typically a bad idea, this works just fine with rootless podman and, more recently, rootless docker. Allowing this is the whole point of user namespaces, after all! Maybe their custom procfs does a better job of pretending to be a VM - but it's simply false that you can't do these things without. You can certainly run a full OS inside Kata/Firecracker, too, I've actually done that. Nitpicking over rating scales aside, the claim that their solution offers large security improvements over any other solution with user namespaces isn't true and the whole thing seems very marketing-driven. The isolation offered by user namespaces is still very weak and not comparable to gVisor or Firecracker (both in production use by Google/AWS for untrusted workloads!). False marketing is a big red flag, especially for something as critical as a container runtime. Anyone who wants unprivileged system containers might want to look into rootless docker or podman rather than this. [1]: https://www.nestybox.com https://www.nestybox.com [2]: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
- lox 5y ago
  I don't spend a lot of time on those comparison-style charts if I'm honest, but that is good (and valid) feedback for them. I also hadn't heard of it, I discovered sysbox via jpettazo's updated post at https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-d..., he's an advisor of nestybox the company that develops sysbox. For the CI/CD usecase on AWS, sysbox presented the right balance of trade-offs between something like Firecracker (which would require bare metal hosts on AWS) and the docker containers that already existed. We specifically need to run privileged containers so that we could run docker-in-docker for CI workloads, so rootless docker or podman wouldn't have helped. Sysbox lets us do that with a significant improvement in security to just running privileged docker containers as most CI environments end up doing. Just switching their docker-in-docker CI job containers to sysbox would have mitigated 4 of the compromises from the article with nearly zero other configuration changes.

by lox 5y ago

xmodem 5y ago

Could you elaborate a bit more how you get the containers into their own IAM roles?

lox 5y ago
Yup, we have a sidecar process/container that runs for each job and assumes an AWS IAM Role for that specific pipeline (with constraints like whether it’s an approved PR as well). The credentials are provided to the job container via a volume mount. This allows us to have shared agents with very granular roles per-pipeline and job.
nijave 5y ago
Not sure if this applies to the parent, but one way this Buildkite Queues map pipelines to agents. Agents can be assigned IAM roles. If you want a certain build to run as an IAM role, you give it a queue where the agents have that role. For AWS, Buildkite has as a Cloud Formation stack that sets up auto scaling groups and some other resources for your agents to run.
- xmodem 5y ago
  Most CI systems will have some way of assigning builds to groups of agents. But it would in some cases be useful to grant different privileges to different containers running on the same agent, which is what I understood OP to have.
orf 5y ago
AWS has IAM service accounts for containers. Comes for free with EKS, not sure how you’d do it without EKS. Basically it adds a signed web identity file into the container which can be used to assume roles.
- captn3m0 5y ago
  Ref: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f...
- otterley 5y ago
  Amazon ECS also offers task roles which do the same thing: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
selecsosi 5y ago
For ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

lima 5y ago

Never head of Sysbox before. At a first glance, the comparison table in their GitHub repo and on their website[1] has a number of inaccuracies which makes me question the quality of their engineering: — They claim that their solution has the same isolation level ("4 stars") than gVisor, unlike "standard containers", which are "2 stars" only (with Firecracker and Kubevirt being "5 stars). This is very wrong - as far as I can tell, they use regular Linux namespaces with some light eBPF-based filesystem emulation, while the vast majority of syscalls is still handled by the host kernel. Sorry, but this is still "2 stars" and far away from the isolation guarantees provided by gVisor (fully emulating the kernel in userspace, which is at the same level or even better than Firecracker) and nowhere close to a VM. — Somehow, regular VMs (Kubevirt) get a "speed" rating of only "2 stars" - worse than gVisor ("3 stars") and Firecracker ("4 stars"), even though they both rely on virtually the same virtualization technology. If anything, gVisor is the slowest but most efficient solution while QEMU maintains some performance advantage over Firecracker[2]. These are basically random scores, it's not a good first impression–if you do a detailed comparison like that, at least do a proper evaluation before giving your own product the best score! — They claim that "standard containers" cannot run a full OS. This isn't true - while it's typically a bad idea, this works just fine with rootless podman and, more recently, rootless docker. Allowing this is the whole point of user namespaces, after all! Maybe their custom procfs does a better job of pretending to be a VM - but it's simply false that you can't do these things without. You can certainly run a full OS inside Kata/Firecracker, too, I've actually done that. Nitpicking over rating scales aside, the claim that their solution offers large security improvements over any other solution with user namespaces isn't true and the whole thing seems very marketing-driven. The isolation offered by user namespaces is still very weak and not comparable to gVisor or Firecracker (both in production use by Google/AWS for untrusted workloads!). False marketing is a big red flag, especially for something as critical as a container runtime. Anyone who wants unprivileged system containers might want to look into rootless docker or podman rather than this. [1]: https://www.nestybox.com https://www.nestybox.com [2]: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf https://www.usenix.org/system/files/nsdi20-paper-agache.pdf

lox 5y ago
I don't spend a lot of time on those comparison-style charts if I'm honest, but that is good (and valid) feedback for them. I also hadn't heard of it, I discovered sysbox via jpettazo's updated post at https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-d..., he's an advisor of nestybox the company that develops sysbox. For the CI/CD usecase on AWS, sysbox presented the right balance of trade-offs between something like Firecracker (which would require bare metal hosts on AWS) and the docker containers that already existed. We specifically need to run privileged containers so that we could run docker-in-docker for CI workloads, so rootless docker or podman wouldn't have helped. Sysbox lets us do that with a significant improvement in security to just running privileged docker containers as most CI environments end up doing. Just switching their docker-in-docker CI job containers to sysbox would have mitigated 4 of the compromises from the article with nearly zero other configuration changes.