5 ms·
Is the 100 MB model being downloaded from HuggingFace on Fly too? I ask this because Fly has immutable Docker containers which wouldn't store any data unless y
by timenova 3y ago
Is the 100 MB model being downloaded from HuggingFace on Fly too?
I ask this because Fly has immutable Docker containers which wouldn't store any data unless you use Fly Volumes. So it could be that Fly is downloading the 100MB model each time it cold-boots.
If that's the case, a multi-stage Dockerfile could help in bundling the model in, and perhaps reducing cold-boot time even further.
- Szpadel 3y agowhy multi-stage would make any difference? isn't multi stage just implementation detail for resulting image? AFAIK only build caching could benefit, but pulling image should be unaltered with or without using multi-stage build
- timenova 3y agoI am guessing that it would be possible to download the model from HuggingFace in the build step. I'm not sure though. Plus the Cog image is 14GB in size (mentioned in the article), I hope there's a way to reduce that.
- Szpadel 3y agoSure, but I'm still do not understand why multi-stage should make any difference, because at the end it's just flattened as layer from copy between stages. So the problem of size is exactly the same, and RUN with curl will have identical size as COPY layer from COPY --from=stage Am I missing something? I can see only benefit for build cache reuse, so download is independed from building code so you won't redownload 14G when you change code, is that what you had in mind?
- simonw 3y agoFly recommend you use one of their mountable volumes for large model files rather than building it into the Docker image - you get faster cold starts that way, plus I think the images have size limits.
- venkii 3y agoYes, correct the 100MB is being downloaded on every boot! I tested it initially because it’s the naivest implementation. The right implementation would bundle it in. But I ended up primarily reporting timings that stop counting up as soon as control is handed over to user generated code - since that’s the number you care about the most.
- timenova 3y agoThat's fair. Perhaps a good future idea would be to benchmark between bundling it in the Docker image, vs. using Fly Volumes as Simon suggested in a sibling comment.