4 ms·

Gvisor on Raspbian

76 points by _ananos_ 7mo ago

geerlingguy 7mo ago
> Fair warning: compiling a kernel on the Pi itself takes several hours. One nit: this should only take about 40 minutes on a Pi 5, assuming you're compiling with -j6 to use all the cores. (Still faster to cross-compile)
- _ananos_ 7mo ago
  well, the tricky detail here (which we do not mention in the post, our bad) is that we got the raspbian config (cp /boot/config ... .config && make oldconfig) which includes most modules, and that's why it took more. But yeap, good point about using the -j flag, it really accelerates the build!
- bionade24 7mo ago
  Using distcc networked compilation instead of cross-compiling is reasonably fast too and easier to set up if one isn't familiar with either.
  - uber1024 7mo ago
    the most frustrating part with having to compile a custom kernel is the maintenance burden (packaging/updating etc.), and not the time it takes to build… I had a similar issue with networking modules for calico (k8s cni) on both rpis and jetson boards…
- pelcg 7mo ago
  That is kind of what I was thinking too, and cross-compilation is still the fastest way to build for a different target.
pelcg 7mo ago
What use-cases are there for gVisor on Raspbian, given that the target is a Raspberry Pi?
- _ananos_ 7mo ago
  the simplest one (and the one we're targetting) is multi-tenant services. You want to sandbox your service so that it doesn't affect the rest of the services running. <shameless plug> We're building a container runtime to do this, and we are comparing alternatives, that's how we got there: https://github.com/urunc-dev/urunc</shameless plug>
  - 0x457 7mo ago
    > given that the target is a Raspberry Pi? Why one would use gVisor is clear, but why would one do that in RPi?
    - _ananos_ 7mo ago
      a number of reasons -- power budget, form factor, experimenting as a testbed for more "elaborate" setups (like robotics combined with a low-end TPU like the coral, or a jetson nano) consider that you can take advantage of all the cloud-native goodies, all wrapped up in a 10x5 box with 5-10W (or 25-30W if you consider jetson boards).
Lliora 7mo ago
Ran gVisor on a Pi 4 cluster for home IoT sandboxing. Memory overhead is real—about 120MB per sandbox vs 15MB for raw containers. On 4GB boards that limits you to ~25 isolated services before OOM kicks in. Also, syscall拦截 adds 30-40% CPU overhead on ARM. Works fine for untrusted Python scripts, but I wouldn’t run anything compute-heavy.
- eptcyka 7mo ago
  Wouldn’t compute workloads be fine as they should not be syscall bound?
  - _ananos_ 7mo ago
    yeap -- compute would be nearly the same. I suspect you need some kind of I/O to make your compute useful (get input for the computation / produce output etc.) so, still, this would have a negative effect overall.
bitwize 7mo ago
gVisor's architecture is fascinating. Years ago I wrote an essay that was kind of a response to all the cracks about "systemd-kerneld" that have been made over the years. Written in character as "Fake Lennart Poettering", it proposed a strategy, using techniques inspired by NetBSD's rump kernel libraries, to turn systemd into a kernel, which would then load a Linux image and "pass through" all system calls to it except those systemd wanted to intercept. Which is kinda the opposite of what gVisor does! Some guy on a German Linux forum thought my idea was an actual plan by the systemd team, and another poster had to correct him that the author was "FAKE Lennart Poettering", so the joke either didn't land or landed all too well, depending on your perspective...
- _ananos_ 7mo ago
  well, jokes aside, what you're describing, is kind of what a "secure" (with many air/literal quotes) MCP/Agentic architecture looks like :D In this context we're experimenting with gvisor on various platforms, and we're preparing a demo for kubecon about a fine-grained sandboxing approach for AI agent tasks spawned from a sandboxed agent.
tetris11 7mo ago
Is proot related to Gvisor?
- _ananos_ 7mo ago
  wasn't familiar with proot -- with a quick look I think proot is a fancy chroot -- which, in turn, is kind of "the first step" for a generic container. to achieve the isolation that gvisor offers you would have to intercept syscalls, create a separate mount/user/net namespace etc. regardless, I don't think proot is somehow related to gvisor ;)
  - tetris11 7mo ago
    It does though, it has user-space implementations of chroot, mount and kernel syscalls. You can even run a debian image built with a later kernel on an older linux system

76 points by _ananos_ 7mo ago

> Fair warning: compiling a kernel on the Pi itself takes several hours. One nit: this should only take about 40 minutes on a Pi 5, assuming you're compiling with -j6 to use all the cores. (Still faster to cross-compile)

_ananos_ 7mo ago
well, the tricky detail here (which we do not mention in the post, our bad) is that we got the raspbian config (cp /boot/config ... .config && make oldconfig) which includes most modules, and that's why it took more. But yeap, good point about using the -j flag, it really accelerates the build!
bionade24 7mo ago
Using distcc networked compilation instead of cross-compiling is reasonably fast too and easier to set up if one isn't familiar with either.
- uber1024 7mo ago
  the most frustrating part with having to compile a custom kernel is the maintenance burden (packaging/updating etc.), and not the time it takes to build… I had a similar issue with networking modules for calico (k8s cni) on both rpis and jetson boards…
pelcg 7mo ago
That is kind of what I was thinking too, and cross-compilation is still the fastest way to build for a different target.

pelcg 7mo ago

What use-cases are there for gVisor on Raspbian, given that the target is a Raspberry Pi?

_ananos_ 7mo ago
the simplest one (and the one we're targetting) is multi-tenant services. You want to sandbox your service so that it doesn't affect the rest of the services running. <shameless plug> We're building a container runtime to do this, and we are comparing alternatives, that's how we got there: https://github.com/urunc-dev/urunc</shameless plug>
- 0x457 7mo ago
  > given that the target is a Raspberry Pi? Why one would use gVisor is clear, but why would one do that in RPi?
  - _ananos_ 7mo ago
    a number of reasons -- power budget, form factor, experimenting as a testbed for more "elaborate" setups (like robotics combined with a low-end TPU like the coral, or a jetson nano) consider that you can take advantage of all the cloud-native goodies, all wrapped up in a 10x5 box with 5-10W (or 25-30W if you consider jetson boards).

Lliora 7mo ago

Ran gVisor on a Pi 4 cluster for home IoT sandboxing. Memory overhead is real—about 120MB per sandbox vs 15MB for raw containers. On 4GB boards that limits you to ~25 isolated services before OOM kicks in. Also, syscall拦截 adds 30-40% CPU overhead on ARM. Works fine for untrusted Python scripts, but I wouldn’t run anything compute-heavy.

eptcyka 7mo ago
Wouldn’t compute workloads be fine as they should not be syscall bound?
- _ananos_ 7mo ago
  yeap -- compute would be nearly the same. I suspect you need some kind of I/O to make your compute useful (get input for the computation / produce output etc.) so, still, this would have a negative effect overall.

bitwize 7mo ago

gVisor's architecture is fascinating. Years ago I wrote an essay that was kind of a response to all the cracks about "systemd-kerneld" that have been made over the years. Written in character as "Fake Lennart Poettering", it proposed a strategy, using techniques inspired by NetBSD's rump kernel libraries, to turn systemd into a kernel, which would then load a Linux image and "pass through" all system calls to it except those systemd wanted to intercept. Which is kinda the opposite of what gVisor does! Some guy on a German Linux forum thought my idea was an actual plan by the systemd team, and another poster had to correct him that the author was "FAKE Lennart Poettering", so the joke either didn't land or landed all too well, depending on your perspective...

_ananos_ 7mo ago
well, jokes aside, what you're describing, is kind of what a "secure" (with many air/literal quotes) MCP/Agentic architecture looks like :D In this context we're experimenting with gvisor on various platforms, and we're preparing a demo for kubecon about a fine-grained sandboxing approach for AI agent tasks spawned from a sandboxed agent.

tetris11 7mo ago

Is proot related to Gvisor?

_ananos_ 7mo ago
wasn't familiar with proot -- with a quick look I think proot is a fancy chroot -- which, in turn, is kind of "the first step" for a generic container. to achieve the isolation that gvisor offers you would have to intercept syscalls, create a separate mount/user/net namespace etc. regardless, I don't think proot is somehow related to gvisor ;)
- tetris11 7mo ago
  It does though, it has user-space implementations of chroot, mount and kernel syscalls. You can even run a debian image built with a later kernel on an older linux system