5 ms·
Cloudflare has almost no cold boot and I think their ml models are prefetched within the same DC, so loading the model the first time should have no noticable o
by NicoJuicy 3y ago
Cloudflare has almost no cold boot and I think their ml models are prefetched within the same DC, so loading the model the first time should have no noticable overhead either.
Correct me if I'm wrong, but that's how I interpreted it when they first started with ML models on their CPU's.
- celso 3y agoThe models in our catalog are all pre-loaded before requests come in.
- lampington 3y agoMakes sense. A bring-your-own-model feature would be awesome, but would make it impossible to do that for rarely-used models without using valuable GPU RAM.