3 ms·

Are there any open source solutions or is cerebrium open-sourcing the technology behind it. Although I liked the read-up (side note: you might have to tone down

Are there any open source solutions or is cerebrium open-sourcing the technology behind it. Although I liked the read-up (side note: you might have to tone down the animations, as others have said, it was a bit dizzy) but overall it was a nice read but I still wish for more technical details as Cold starts are something that just is something that I am interested in. So are there any more resources that perhaps the team could point out or other resources or if there are any idea of open-sourcing it ever for more internal deeper dives as I would love to know more about it!

eperot 3mo ago
gVisor is open-source, and `cuda-checkpoint` is freely available. gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you specify a program to execute before gVisor starts taking the process snapshot. You can fill in the blanks from there.
- za_mike157 3mo ago
  Us and the team from Modal have been upstreaming things to the GVisor repo (https://github.com/google/gvisor/pulls https://github.com/google/gvisor/pulls) in order to make it compatible with cuda-checkpoint and other parts of our system. While we are both contributing fixes and performance improvements we are unfortunately leaving some secret sauce on the side but hopefully it should get most folks to a successful implementation as is
  - ranjanayush 3mo ago
    Recently gVisor added cuda checkpoint automation support, please consider using that: https://github.com/google/gvisor/commit/b3c184ce6b808648ddcaed4e59744200015d8c0b https://github.com/google/gvisor/commit/b3c184ce6b808648ddca...
    - za_mike157 3mo ago
      Thanks for this! We did see it and it was committed long after we already implemented this functionality. However, there are a lot of edge cases that this commit doesn't deal with that users are going to have to do
harleyjs 3mo ago
This sort of technology is available on GKE https://docs.cloud.google.com/kubernetes-engine/docs/concepts/pod-snapshots https://docs.cloud.google.com/kubernetes-engine/docs/concept...
- za_mike157 3mo ago
  Interesting! I didn't see they released this. Do you know what their benchmarks are? I know for cloud run they are pretty slow

eperot 3mo ago

gVisor is open-source, and `cuda-checkpoint` is freely available. gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you specify a program to execute before gVisor starts taking the process snapshot. You can fill in the blanks from there.

za_mike157 3mo ago
Us and the team from Modal have been upstreaming things to the GVisor repo (https://github.com/google/gvisor/pulls https://github.com/google/gvisor/pulls) in order to make it compatible with cuda-checkpoint and other parts of our system. While we are both contributing fixes and performance improvements we are unfortunately leaving some secret sauce on the side but hopefully it should get most folks to a successful implementation as is
- ranjanayush 3mo ago
  Recently gVisor added cuda checkpoint automation support, please consider using that: https://github.com/google/gvisor/commit/b3c184ce6b808648ddcaed4e59744200015d8c0b https://github.com/google/gvisor/commit/b3c184ce6b808648ddca...
  - za_mike157 3mo ago
    Thanks for this! We did see it and it was committed long after we already implemented this functionality. However, there are a lot of edge cases that this commit doesn't deal with that users are going to have to do

harleyjs 3mo ago

This sort of technology is available on GKE https://docs.cloud.google.com/kubernetes-engine/docs/concepts/pod-snapshots https://docs.cloud.google.com/kubernetes-engine/docs/concept...

za_mike157 3mo ago
Interesting! I didn't see they released this. Do you know what their benchmarks are? I know for cloud run they are pretty slow