3 ms·
Curious about what HN things about llamafile and modelfile (https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md https://github.com/jmorganca/ollama/
by verdverm 3y ago
Curious about what HN things about llamafile and modelfile (https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md https://github.com/jmorganca/ollama/blob/main/docs/modelfile...)
Both invoke a Dockerfile like experience. Modelfile immediately seems like a Dockerfile, but llamafile looks harder to use. It is not immediately clear what it looks like. Is it a sequence of commands at the terminal?
My theory question is, why not use a Dockerfile for this?
- airstrike 3y agohttps://news.ycombinator.com/item?id=38464057 https://news.ycombinator.com/item?id=38464057
- verdverm 3y agooh right, one of the things llamafile can do is run on windows directly, and generally without the extra need for docker and such
- throwup238 3y agoTheir killer feature is the --grammar option which restricts the logits the LLM outputs which makes them great for bash scripts that do all manner of NLP classification work. Otherwise I use ollama when I need a local LLM, vllm when I'm renting GPU servers, or OpenAI API when I just want the best model.
- verdverm 3y agoInteresting, you inspired me to https://github.com/jmorganca/ollama/issues/1507 https://github.com/jmorganca/ollama/issues/1507 I've also had good success instructing arbitrary grammars in the system prompt, though it doesn't work at the logits level, which can be helpful
- throwup238 3y agoThere's an old PR for it: https://github.com/jmorganca/ollama/pull/565 https://github.com/jmorganca/ollama/pull/565 (it just uses the underlying llama.cpp grammar feature which is what llamafile does)
- verdverm 3y agothanks for the pointer, I'll update my issue to reference that PR
- jart 3y agollamafile doesn't use docker. It's just an executable file. You download your llamafile, chmod +x it, then you ./run it. It is possible to use Docker + llamafile though. Check out https://github.com/ajbouh/cosmos https://github.com/ajbouh/cosmos which has a pretty good Dockerfile configuration you can use.
- verdverm 3y agoI already use containers for all my current AI stuff, so I don't really need the llamafile part. The question was more about alternatives, because on the surface, it looks to have a lot of overlap with containers. The main difference I've seen is that llamafiles are easier if you do not already use containers or are on a platform where containers come with a lot of overhead and limitations
- lucb1e 3y agoWait, I think I'm misunderstanding. It feels like you're asking what you need a computer for if you've already got a web browser, whereas one is a component necessary for the other. Inside of those containers you're using, there's executables. If it runs as a nice, self-contained single binary that you can just call, why still look for a container to wrap around it and invoke using a more complicated command? How is running a binary in a container an alternative to running said binary?
- verdverm 3y ago> Wait, I think I'm misunderstanding. Yes, and you are being snarky about it I already have an AI development workflow and production environment based on containers. Technically, not ML executables inside the container, rather a Python runtime and code. This also comes with an ecosystem for things you need in real work that are general beyond AI applications. Why would I want to add additional tooling and workflow that only works locally and needs all the extras to be added on?
- simonw 3y ago