3 ms·
Really interesting work. we’ve been building a container-native snapshotting system too, but focused on cold start reduction and multi-model orchestration for
by pveldandi 1y ago
Really interesting work. we’ve been building a container-native snapshotting system too, but focused on cold start reduction and multi-model orchestration for LLM inference.
Different use case (sub-2s loading for large models), but very similar challenges around memory, device state, and restore reliability.
- qnib 1y agoRadostin from RedHat did some interesting work on hot-swapping (kinda cold-start) of models: https://www.researchgate.net/publication/392759373_Transparent_Hot-Swapping_of_Containerized_AIML_Workloads https://www.researchgate.net/publication/392759373_Transpare... Definitely something that might be added in the future.