7 ms·

Barco: Linux Containers from Scratch in C

287 points by lucavallin 3y ago

lucavallin 3y ago
barco is a project I worked on to learn more about Linux containers and the Linux kernel, based on other guides on the internet.
- intelVISA 3y ago
  awesome, thx for sharing this :)
- metadat 3y ago
  How did you come up with the name "Barco"?
  - voidmain0001 3y ago
    It’s Spanish for boat.
  - 29athrowaway 3y ago
    docker has to do with ships and barco means ship.
    - ofrzeta 3y ago
      It's funny that three people feel the urge to reply for the author and all of them are wrong :-) Also I find it interesting that in Portuguese as well as in Spanish the word exists in both genders: el barco, la barca, o barco, a barca.
      - darkwater 3y ago
        At least in Spanish, my rule of thumb is that "barco" is for bigger boats and "barca" is for smaller ones (and then you have Barça which is the football/soccer team).
        loloquwowndueo 3y ago
        Barça looks similar but is pronounced entirely differently (say “barsa”).
      - quickthrower2 3y ago
        It is wrong but coincidently in a way that aligns with the sea-fairing theme
  - lcvln 3y ago
    It's Venetian (my native language) for "hay barrack": http://vec.wikipedia.org/wiki/Barco http://vec.wikipedia.org/wiki/Barco
  - mlashcorp 3y ago
    Also, Portuguese for boat.
- murphyslaw 3y ago
  This could be quite useful in a CS course on containers.
- teleforce 3y ago
  Looks like a good project to learn container from scratch. Just wondering the main reason you're C since most of the container project now seems to be using Go or Rust?
  - cyphar 3y ago
    As the maintainer of a Go container runtime (runc), and having worked with Rust in various other projects, while they can be better languages for building large projects, they make it harder to understand what exactly your program is doing when writing software like this. One example that immediately comes to mind from Rust is a bug with O_PATH file descriptors I found a while ago[1], which would've made certain code we use in runc not work. And from Go, here is a bug I just found in their code for handling file descriptors for ForkExec[2] which is causing issues in a runc patch I'm working on. Neither of these issues exist in C programs. Though of course, C programs have their own issues. For better or worse, the Linux kernel APIs are easiest to use from C. In runc we actually implement the core container setup code in C because Go doesn't allow you to do everything we need for setting up a container (it has gotten better though, in the past it was completely impossible to set up a container properly in pure Go -- now you can set one up but there are still certain configurations that are not possible to implement in pure Go, such as "docker exec"). You also cannot run Go in single-threaded mode, which means that certain kernel APIs (unshare(CLONE_NEWUSER) for instance) simply cannot be used from regular Go code. [1]: https://github.com/rust-lang/rust/issues/62314 https://github.com/rust-lang/rust/issues/62314 [2]: https://github.com/golang/go/issues/61751 https://github.com/golang/go/issues/61751
    - bheadmaster 3y ago
      Wouldn't "runtime.LockOSThead()" help with the single-threaded API?
      - cyphar 3y ago
        No, that only pins the current goroutine to a single OS thread (which is needed for some APIs -- namely, all of the other namespace APIs and some thread-related APIs). There is no way to make an entire Go program run as a single threaded program without using CGo the way we do in runc. Even GOMAXPROCS=1 doesn't work. CLONE_NEWUSER will always fail in a multi-threaded program.
gjkood 3y ago
[flagged]
- CameronNemo 3y ago
  Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
  - Philpax 3y ago
    GP was asking if it could cause an issue for the OP, not complaining about an annoyance. It's something that the OP may want to address.
    - CameronNemo 3y ago
      I still think it is tangential. The author stated that they wrote this project to learn. The readme says that it is not intended for production use and that there is no networking set up in the containers. With that context, I doubt that name collisions outside of the containers space are top of mind.
      - lcvln 3y ago
        barco really just means "hay barrack" in my native language ¯\_(ツ)_/¯
        brezelgoring 3y ago
        It means 'watercraft' or 'sea worthy ship' in Spanish, too. :) In what language does it mean 'hay barrack'?
        CameronNemo 3y ago
        Indeed the wiki yields no matches https://en.m.wiktionary.org/wiki/barco https://en.m.wiktionary.org/wiki/barco
shortrounddev2 3y ago
Very cool, I was thinking of doing something similar in windows
- semiquaver 3y ago
  What is the underlying isolation technology that would be used in windows?
  - AkshitGarg 3y ago
    Windows also supports containers: https://learn.microsoft.com/en-us/virtualization/windowscontainers/about/ https://learn.microsoft.com/en-us/virtualization/windowscont...
    - shortrounddev2 3y ago
      It also has sandboxing with app containers
zamalek 3y ago
> barco enforces a minimal set of restrictions to run untrusted code, which is not recommended for production use, where a more robust solution should be used. Aren't containers never suitable for running untrusted code? You need AppArmor, bwrap, or similar AFAIK.
- loeg 3y ago
  I would probably point at a virtual machine for a convenient place to run untrusted code. It's not perfect -- there are VM escapes -- but it's more convenient than a dedicated, air-gapped machine.
  - CameronNemo 3y ago
    GKE runs every kubelet in its own gvisor-like userspace hypervisor. https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 https://cloud.google.com/blog/products/containers-kubernetes...
- charcircuit 3y ago
  >Aren't containers never suitable for running untrusted code? They are suitable provided the kernel is secure.
  - cyphar 3y ago
    This is tautologically true -- "Is X secure? Yes, assuming the technology X uses is secure." The more nuanced answer is that containers have several layers of protections (seccomp, LSMs, user namespaces, namespaces, cgroups, capabilities, and standard process permissions by running as an unprivileged user) which all act together to help protect against container attacks. It's not perfect, but most container breakout attacks we've had so far are related to when container runtimes have to operate on a container during process setup (IMHO because the process for creating a container process is far from atomic) -- some of these attacks were enabled by kernel bugs which we went and fixed as well. It is very difficult to break out of a container once it has been configured and left alone.
godlover 3y ago
?
favflam 3y ago
Question: for sandboxing untrusted code, should I invest time in learning more linux container stuff or switch to learning WASI? I am inclined towards WASI myself.
- lmm 3y ago
  I have more faith in WASI. Linux containers is inherently in a whack-a-mole position where they're trying to retrofit security onto something that was never built for it, which very rarely works.
- jeroenhd 3y ago
  There's a huge performance hit for many programming languages when you run them inside a WASM runtime. Memory also behaves very different from normal applications. Autovectorizasation also isn't universally supported by WASM compilers yet, which can be costly for performance. Properly configured WASI runtime are great for security but they're worse than containers on most other fronts. I don't think the downsides make sense unless you're building a business that lets random customers upload WASM files you execute.
yukIttEft 3y ago
Could this also be implemented as a bashscript?
- lcvln 3y ago
  Probably: https://github.com/p8952/bocker https://github.com/p8952/bocker
nedt 3y ago
When I did a talk about docker I also wanted to show a bit of what it does under the hood without going through all the layers and without too much details. This ~120 lines of shell script is really good in providing just an intro into what's needed for containers: https://github.com/p8952/bocker/blob/master/bocker https://github.com/p8952/bocker/blob/master/bocker (not mine)
elcapitan 3y ago
Just came to say that this is a very nicely set up C project, understandable structure and simple Makefile, very beginner friendly, congrats :)
hippospark 3y ago
There is a similar project _Linux containers in 500 lines of code_ [1], the code is a bit old, but the procedure is quite simple. [1]: https://blog.lizzie.io/linux-containers-in-500-loc.html https://blog.lizzie.io/linux-containers-in-500-loc.html

287 points by lucavallin 3y ago

barco is a project I worked on to learn more about Linux containers and the Linux kernel, based on other guides on the internet.

intelVISA 3y ago
awesome, thx for sharing this :)
metadat 3y ago
How did you come up with the name "Barco"?
- voidmain0001 3y ago
  It’s Spanish for boat.
- 29athrowaway 3y ago
  docker has to do with ships and barco means ship.
  - ofrzeta 3y ago
    It's funny that three people feel the urge to reply for the author and all of them are wrong :-) Also I find it interesting that in Portuguese as well as in Spanish the word exists in both genders: el barco, la barca, o barco, a barca.
    - darkwater 3y ago
      At least in Spanish, my rule of thumb is that "barco" is for bigger boats and "barca" is for smaller ones (and then you have Barça which is the football/soccer team).
      - loloquwowndueo 3y ago
        Barça looks similar but is pronounced entirely differently (say “barsa”).
    - quickthrower2 3y ago
      It is wrong but coincidently in a way that aligns with the sea-fairing theme
- lcvln 3y ago
  It's Venetian (my native language) for "hay barrack": http://vec.wikipedia.org/wiki/Barco http://vec.wikipedia.org/wiki/Barco
- mlashcorp 3y ago
  Also, Portuguese for boat.
murphyslaw 3y ago
This could be quite useful in a CS course on containers.
teleforce 3y ago
Looks like a good project to learn container from scratch. Just wondering the main reason you're C since most of the container project now seems to be using Go or Rust?
- cyphar 3y ago
  As the maintainer of a Go container runtime (runc), and having worked with Rust in various other projects, while they can be better languages for building large projects, they make it harder to understand what exactly your program is doing when writing software like this. One example that immediately comes to mind from Rust is a bug with O_PATH file descriptors I found a while ago[1], which would've made certain code we use in runc not work. And from Go, here is a bug I just found in their code for handling file descriptors for ForkExec[2] which is causing issues in a runc patch I'm working on. Neither of these issues exist in C programs. Though of course, C programs have their own issues. For better or worse, the Linux kernel APIs are easiest to use from C. In runc we actually implement the core container setup code in C because Go doesn't allow you to do everything we need for setting up a container (it has gotten better though, in the past it was completely impossible to set up a container properly in pure Go -- now you can set one up but there are still certain configurations that are not possible to implement in pure Go, such as "docker exec"). You also cannot run Go in single-threaded mode, which means that certain kernel APIs (unshare(CLONE_NEWUSER) for instance) simply cannot be used from regular Go code. [1]: https://github.com/rust-lang/rust/issues/62314 https://github.com/rust-lang/rust/issues/62314 [2]: https://github.com/golang/go/issues/61751 https://github.com/golang/go/issues/61751
  - bheadmaster 3y ago
    Wouldn't "runtime.LockOSThead()" help with the single-threaded API?
    - cyphar 3y ago
      No, that only pins the current goroutine to a single OS thread (which is needed for some APIs -- namely, all of the other namespace APIs and some thread-related APIs). There is no way to make an entire Go program run as a single threaded program without using CGo the way we do in runc. Even GOMAXPROCS=1 doesn't work. CLONE_NEWUSER will always fail in a multi-threaded program.

gjkood 3y ago

[flagged]

CameronNemo 3y ago
Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
- Philpax 3y ago
  GP was asking if it could cause an issue for the OP, not complaining about an annoyance. It's something that the OP may want to address.
  - CameronNemo 3y ago
    I still think it is tangential. The author stated that they wrote this project to learn. The readme says that it is not intended for production use and that there is no networking set up in the containers. With that context, I doubt that name collisions outside of the containers space are top of mind.
    - lcvln 3y ago
      barco really just means "hay barrack" in my native language ¯\_(ツ)_/¯
      - brezelgoring 3y ago
        It means 'watercraft' or 'sea worthy ship' in Spanish, too. :) In what language does it mean 'hay barrack'?
        CameronNemo 3y ago
        Indeed the wiki yields no matches https://en.m.wiktionary.org/wiki/barco https://en.m.wiktionary.org/wiki/barco

shortrounddev2 3y ago

Very cool, I was thinking of doing something similar in windows

semiquaver 3y ago
What is the underlying isolation technology that would be used in windows?
- AkshitGarg 3y ago
  Windows also supports containers: https://learn.microsoft.com/en-us/virtualization/windowscontainers/about/ https://learn.microsoft.com/en-us/virtualization/windowscont...
  - shortrounddev2 3y ago
    It also has sandboxing with app containers

zamalek 3y ago

> barco enforces a minimal set of restrictions to run untrusted code, which is not recommended for production use, where a more robust solution should be used. Aren't containers never suitable for running untrusted code? You need AppArmor, bwrap, or similar AFAIK.

loeg 3y ago
I would probably point at a virtual machine for a convenient place to run untrusted code. It's not perfect -- there are VM escapes -- but it's more convenient than a dedicated, air-gapped machine.
- CameronNemo 3y ago
  GKE runs every kubelet in its own gvisor-like userspace hypervisor. https://cloud.google.com/blog/products/containers-kubernetes/how-gvisor-protects-google-cloud-services-from-cve-2020-14386 https://cloud.google.com/blog/products/containers-kubernetes...
charcircuit 3y ago
>Aren't containers never suitable for running untrusted code? They are suitable provided the kernel is secure.
- cyphar 3y ago
  This is tautologically true -- "Is X secure? Yes, assuming the technology X uses is secure." The more nuanced answer is that containers have several layers of protections (seccomp, LSMs, user namespaces, namespaces, cgroups, capabilities, and standard process permissions by running as an unprivileged user) which all act together to help protect against container attacks. It's not perfect, but most container breakout attacks we've had so far are related to when container runtimes have to operate on a container during process setup (IMHO because the process for creating a container process is far from atomic) -- some of these attacks were enabled by kernel bugs which we went and fixed as well. It is very difficult to break out of a container once it has been configured and left alone.

godlover 3y ago

favflam 3y ago

Question: for sandboxing untrusted code, should I invest time in learning more linux container stuff or switch to learning WASI? I am inclined towards WASI myself.

lmm 3y ago
I have more faith in WASI. Linux containers is inherently in a whack-a-mole position where they're trying to retrofit security onto something that was never built for it, which very rarely works.
jeroenhd 3y ago
There's a huge performance hit for many programming languages when you run them inside a WASM runtime. Memory also behaves very different from normal applications. Autovectorizasation also isn't universally supported by WASM compilers yet, which can be costly for performance. Properly configured WASI runtime are great for security but they're worse than containers on most other fronts. I don't think the downsides make sense unless you're building a business that lets random customers upload WASM files you execute.

yukIttEft 3y ago

Could this also be implemented as a bashscript?

lcvln 3y ago
Probably: https://github.com/p8952/bocker https://github.com/p8952/bocker

nedt 3y ago

When I did a talk about docker I also wanted to show a bit of what it does under the hood without going through all the layers and without too much details. This ~120 lines of shell script is really good in providing just an intro into what's needed for containers: https://github.com/p8952/bocker/blob/master/bocker https://github.com/p8952/bocker/blob/master/bocker (not mine)

elcapitan 3y ago

Just came to say that this is a very nicely set up C project, understandable structure and simple Makefile, very beginner friendly, congrats :)

hippospark 3y ago

There is a similar project _Linux containers in 500 lines of code_ [1], the code is a bit old, but the procedure is quite simple. [1]: https://blog.lizzie.io/linux-containers-in-500-loc.html https://blog.lizzie.io/linux-containers-in-500-loc.html