7 ms·
Popped it into a docker setup: https://github.com/tluyben/llamafile-docker https://github.com/tluyben/llamafile-docker to save even more keystrokes.
by tluyben2 3y ago
Popped it into a docker setup:
https://github.com/tluyben/llamafile-docker https://github.com/tluyben/llamafile-docker
to save even more keystrokes.
- luke-stanley 3y agoIt was already great, and this is more great for those who get Docker-Compose or are patient enough to figure out. But if you're gonna have Docker, you could also use bleeding edge llama.cpp with a few more lines too! What a time to be alive innit!
- finnjohnsen2 3y agoThank you kindly
- reisse 3y agoWhat is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
- belter 3y agoDeploy your models into a Kubernetes cluster and let them fight for resources to death? A modern Roman Gladiators circus with Models?
- ukuina 3y agoGive them write access to your control-plane and the first one to write syntactically correct IaC wins!
- Bjartr 3y agoMore like a sequel to Core War... Which actually does sound pretty amusing now that I think about it. Call it Cloud War. https://en.m.wikipedia.org/wiki/Core_War https://en.m.wikipedia.org/wiki/Core_War
- finnjohnsen2 3y agoI see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.
- reisse 3y agoI see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume for model? Cargo cult is not in using Docker, but in using Docker to wrap something already wrapped into a comparable layer of abstraction. Besides, > not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet is purely self-deception. It's not like Docker images are not stored in some folder deep in the filesystem. If anything, it's harder to clean up after Docker than just doing rm -rf on a directory with llamafiles.
- simonw 3y agoIf you want to use Docker then you can go ahead - I don't see anyone claiming that this brand new, highly experimental project should be used by everyone instead of Docker. There are tradeoffs hers. For some people the tradeoff of a single executable file with everything in it compared to setting up a Docker system is worthwhile.
- reisse 3y agoSure. I just question why people want to use both simultaneously.
- xylophile 3y agoDo you routinely download unsigned binaries of unprovable provenance and run them? Because if you do, you might eventually find reason to appreciate the additional isolation that namespaces et al give you very conveniently via Docker (or your favorite alternative).
- oefrha 3y agoWrapping it in Docker makes it harder to access your filesystem without some dedicated jailbreak.
- rudasn 3y agoVolumes maybe? I don't see the issue here.
- czbond 3y agosecurity - i have no idea if an executable will not be a malicious actor to my system. Dockerizing it, and removing access to system files - is the reason. I am not trusting that a tradition virus / etc scanner will find stuff hidden in executable models.
- oefrha 3y agoLike the other comment said, not being able to access fs is a feature.
- devneelpatel 3y agoHomelabbers like me have a docker swarm cluster / k8s cluster so this def helps!
- tluyben2 3y agoPersonally I wanted to see if this llama can generate the required docker files to run itself just from pasting in the instructions from the my parent commenter and asking for docker/docker-compose. It did and it works.
- reisse 3y agoWell that's a good reason I didn't think about, thank you!