5 ms·

I discovered CRIU in this video below (1h) "Container Migration and CRIU Details with Adrian Reber (Red Hat)", it has a live demo and the details about how much

I discovered CRIU in this video below (1h) "Container Migration and CRIU Details with Adrian Reber (Red Hat)", it has a live demo and the details about how much "user space" it really is. Here with the RH podman fork of docker. Since everyone is treating containers as cattle CRIU doesn't seem to get much attention, and might be why a video and not a blog post was my first introduction. https://www.youtube.com/watch?v=-7DgNxyuz_o https://www.youtube.com/watch?v=-7DgNxyuz_o

JeremyNT 2y ago
> Since everyone is treating containers as cattle CRIU doesn't seem to get much attention Yeah, I guess that's probably the reason. If you're engineering your workloads with the idea that the world might "poof" out from under you at any moment you'd never wonder about / reach for something like CRIU. It's a trick that I'd never much thought about, but now that I've learned it exists (so many years late) I find myself wondering about the path not taken here. It feels like it should be incredibly useful... but I can't figure out exactly what I'd want to do with it myself.
- yourapostasy 2y ago
  > ...I find myself wondering about the path not taken here. Check out mainframes and Tandem systems for a peek at that path. Lots of support in those systems for the notion your application’s substrate might suddenly go poof, and you need it to recover from where it left off as instantaneously as possible. It’s expensive.
yencabulator 2y ago
> Since everyone is treating containers as cattle CRIU doesn't seem to get much attention Nah, it's more like "I don't trust that thing to not cause weird behavior in production". VM-level snapshots are standard practice[1] because the abstraction there is right-sized for being able to do that reliably. CRIU isn't, because it's trying to solve a much harder problem. [1]: And even there, beware cloning running memory state, you can get weird interactions from two identical parties trying to talk to the same 3rd service, separated by time. Cloning disk snapshots is much safer, and even there you can screw up because of duplicate machine IDs, crypto keys, nonces, etc.
- __turbobrew__ 2y ago
  The thing with VMs is that there is much more overhead to booting a Linux VM which makes checkpointing much more attractive. For a container running with Linux namespaces/cgroups the container can be started in a few milliseconds. Im sure there are some niche applications for container checkpointing, but I don’t really see the complexity being worth it. Maybe checkpointing some long running batch jobs could save you some money, but you should just make your jobs checkpoint their state to an external store such a ceph or s3 and make the jobs smart enough to load any state from those stores if they are preempted.
  - yencabulator 2y ago
    Firecracker starts running the application in as low as 125ms. Most of the overhead in a "cloud cold start" comes from the cloud infrastructure, not from the virtualization mechanism.
- yourapostasy 2y ago
  Yeah, I’ve always held a soft spot for CRIU, but I don’t see it battle-tested enough to trust big, gnarly closed-source third-part vendor enterprise Java products to run under it. And if I’ve reduced an open source piece of kit’s execution footprint enough to trust it, I’d probably reach for Unikraft with checkpointing to a Persistent Volume before CRIU, which feels like early days of VMWare. Hopefully though, my trepidation is wrong. What is the most complex piece of software others have run under CRIU in production, and for how long?
znpy 2y ago
> Here with the RH podman fork of docker small nit: podman is not a docker fork, it's a completely different codebase written from scratch
- cpuguy83 2y ago
  It is absolutely not from scratch. It is definitely reusing docker code (or at least it used to).

JeremyNT 2y ago

> Since everyone is treating containers as cattle CRIU doesn't seem to get much attention Yeah, I guess that's probably the reason. If you're engineering your workloads with the idea that the world might "poof" out from under you at any moment you'd never wonder about / reach for something like CRIU. It's a trick that I'd never much thought about, but now that I've learned it exists (so many years late) I find myself wondering about the path not taken here. It feels like it should be incredibly useful... but I can't figure out exactly what I'd want to do with it myself.

yourapostasy 2y ago
> ...I find myself wondering about the path not taken here. Check out mainframes and Tandem systems for a peek at that path. Lots of support in those systems for the notion your application’s substrate might suddenly go poof, and you need it to recover from where it left off as instantaneously as possible. It’s expensive.

yencabulator 2y ago

> Since everyone is treating containers as cattle CRIU doesn't seem to get much attention Nah, it's more like "I don't trust that thing to not cause weird behavior in production". VM-level snapshots are standard practice[1] because the abstraction there is right-sized for being able to do that reliably. CRIU isn't, because it's trying to solve a much harder problem. [1]: And even there, beware cloning running memory state, you can get weird interactions from two identical parties trying to talk to the same 3rd service, separated by time. Cloning disk snapshots is much safer, and even there you can screw up because of duplicate machine IDs, crypto keys, nonces, etc.

__turbobrew__ 2y ago
The thing with VMs is that there is much more overhead to booting a Linux VM which makes checkpointing much more attractive. For a container running with Linux namespaces/cgroups the container can be started in a few milliseconds. Im sure there are some niche applications for container checkpointing, but I don’t really see the complexity being worth it. Maybe checkpointing some long running batch jobs could save you some money, but you should just make your jobs checkpoint their state to an external store such a ceph or s3 and make the jobs smart enough to load any state from those stores if they are preempted.
- yencabulator 2y ago
  Firecracker starts running the application in as low as 125ms. Most of the overhead in a "cloud cold start" comes from the cloud infrastructure, not from the virtualization mechanism.
yourapostasy 2y ago
Yeah, I’ve always held a soft spot for CRIU, but I don’t see it battle-tested enough to trust big, gnarly closed-source third-part vendor enterprise Java products to run under it. And if I’ve reduced an open source piece of kit’s execution footprint enough to trust it, I’d probably reach for Unikraft with checkpointing to a Persistent Volume before CRIU, which feels like early days of VMWare. Hopefully though, my trepidation is wrong. What is the most complex piece of software others have run under CRIU in production, and for how long?

znpy 2y ago

> Here with the RH podman fork of docker small nit: podman is not a docker fork, it's a completely different codebase written from scratch

cpuguy83 2y ago
It is absolutely not from scratch. It is definitely reusing docker code (or at least it used to).