4 ms·
Show HN: Checkpoint K8s pods transparently (plain CPU or GPU accelerated) [video]
Hi HN,
I am an early adopter of containers with a background in HPC. From the early days, I’ve tried to merge container tech into the HPC (and HTC) stack. Containers already make packing and deployment easier - especially in AI/ML and data science. How about checkpoint/restore?
Over the last couple of months, we at MemVerge have developed a Kubernetes Operator for transparent checkpointing and restoring, allowing you to use discounted Spot instances for long-running workloads, like bioinformatics workflows or ML training.
Here’s how it works:
- the operator attaches a PVC to your pod
- intercepts the STOP signal to checkpoint the pod
- if the attached PVC contains a checkpoint when the pod is starting over, it will be restored instead of starting from scratch.
Here’s a 2m30s video that demonstrates interrupting a small training workload: https://youtu.be/K9yY6_2255Y https://youtu.be/K9yY6_2255Y
This can be triggered by someone draining the node (e.g., due to an EC2 Spot reclaim), deleting a pod, or another operator acting on its own logic.
Our checkpoint engine captures every aspect of the process tree within the container: memory pages, file descriptors—even TCP connections, if you want us to.
Until recently, it was targeted at CPU use cases only. We’ve now added support for NVIDIA GPUs, with AMD GPUs coming soon (via upstream CRIU plugins).
I’ve done some typical checkpoint/restore work (e.g., Jupyter notebooks, traditional jobs) and would love to hear what kinds of workloads you’re interested in checkpointing and restoring.
You can try it out in your Kubernetes environment with our 60-day trial: https://form.typeform.com/to/vZujMYxI https://form.typeform.com/to/vZujMYxI
- pveldandi 1y agoReally interesting work. we’ve been building a container-native snapshotting system too, but focused on cold start reduction and multi-model orchestration for LLM inference. Different use case (sub-2s loading for large models), but very similar challenges around memory, device state, and restore reliability.
- qnib 1y agoRadostin from RedHat did some interesting work on hot-swapping (kinda cold-start) of models: https://www.researchgate.net/publication/392759373_Transparent_Hot-Swapping_of_Containerized_AIML_Workloads https://www.researchgate.net/publication/392759373_Transpare... Definitely something that might be added in the future.
- jwcesign 1y agoSo, this solution is built on CRIU?
- qnib 1y agoessentially yes. The missing piece is experience in making sure the right files, descriptors and memory blobs are saved and brought back in the right order and the coordination with different schedulers (like K8s in this case, but also AWS Batch, HTCondor, SLURM, ...). EC2 Spot only gives you 2min in which you need to wrap up everything, GCP even only 30s. The time pressure is also an obstacle to deal with.