3 ms·

llamafile doesn't use docker. It's just an executable file. You download your llamafile, chmod +x it, then you ./run it. It is possible to use Docker + llamafil

by jart 3y ago

llamafile doesn't use docker. It's just an executable file. You download your llamafile, chmod +x it, then you ./run it. It is possible to use Docker + llamafile though. Check out https://github.com/ajbouh/cosmos https://github.com/ajbouh/cosmos which has a pretty good Dockerfile configuration you can use.

verdverm 3y ago
I already use containers for all my current AI stuff, so I don't really need the llamafile part. The question was more about alternatives, because on the surface, it looks to have a lot of overlap with containers. The main difference I've seen is that llamafiles are easier if you do not already use containers or are on a platform where containers come with a lot of overhead and limitations
- lucb1e 3y ago
  Wait, I think I'm misunderstanding. It feels like you're asking what you need a computer for if you've already got a web browser, whereas one is a component necessary for the other. Inside of those containers you're using, there's executables. If it runs as a nice, self-contained single binary that you can just call, why still look for a container to wrap around it and invoke using a more complicated command? How is running a binary in a container an alternative to running said binary?
  - verdverm 3y ago
    > Wait, I think I'm misunderstanding. Yes, and you are being snarky about it I already have an AI development workflow and production environment based on containers. Technically, not ML executables inside the container, rather a Python runtime and code. This also comes with an ecosystem for things you need in real work that are general beyond AI applications. Why would I want to add additional tooling and workflow that only works locally and needs all the extras to be added on?
    - simonw 3y ago
      It sounds to me like you don't personally need llamafile then.
      - verdverm 3y ago
        yup, trying to understand where all the nascent AI tools are trying to fit in ollama seems way easier to use than llamafile
- baq 3y ago
  well, you can keep using containers if you want. docker sucks really bad on macos, but if you're not using that, you might not care.
  - verdverm 3y ago
    docker is not the only container runtime, no issues using mac as a daily development machine
    - baq 3y ago
      I also tried rancher and it was worse, as in it hung on my workloads, which docker started fine. I had my share of docker configuration wiping and rosetta compatibility fighting, though.
      - verdverm 3y ago
        nerdctl, colima, containerd
- ElectricalUnion 3y ago
  > it looks to have a lot of overlap with containers No it doesn't. They're at different abstraction layers. Containers are more convenient form of generic code/data packaging and isolation primitives. llamafile is the code/data that you want to run. Equivalent poor analogy: Why does JPEG XL exists? I can just put a base64 representation of a PNG image in a data URL inside a HTML file inside a ZIP file; I can also bundle more stuff this way! JPEG XL is useless! No one should use JPEG XL! Other use cases that use JPEG XL by themselves are invalid!

by jart 3y ago

verdverm 3y ago

I already use containers for all my current AI stuff, so I don't really need the llamafile part. The question was more about alternatives, because on the surface, it looks to have a lot of overlap with containers. The main difference I've seen is that llamafiles are easier if you do not already use containers or are on a platform where containers come with a lot of overhead and limitations

lucb1e 3y ago
Wait, I think I'm misunderstanding. It feels like you're asking what you need a computer for if you've already got a web browser, whereas one is a component necessary for the other. Inside of those containers you're using, there's executables. If it runs as a nice, self-contained single binary that you can just call, why still look for a container to wrap around it and invoke using a more complicated command? How is running a binary in a container an alternative to running said binary?
- verdverm 3y ago
  > Wait, I think I'm misunderstanding. Yes, and you are being snarky about it I already have an AI development workflow and production environment based on containers. Technically, not ML executables inside the container, rather a Python runtime and code. This also comes with an ecosystem for things you need in real work that are general beyond AI applications. Why would I want to add additional tooling and workflow that only works locally and needs all the extras to be added on?
  - simonw 3y ago
    It sounds to me like you don't personally need llamafile then.
    - verdverm 3y ago
      yup, trying to understand where all the nascent AI tools are trying to fit in ollama seems way easier to use than llamafile
baq 3y ago
well, you can keep using containers if you want. docker sucks really bad on macos, but if you're not using that, you might not care.
- verdverm 3y ago
  docker is not the only container runtime, no issues using mac as a daily development machine
  - baq 3y ago
    I also tried rancher and it was worse, as in it hung on my workloads, which docker started fine. I had my share of docker configuration wiping and rosetta compatibility fighting, though.
    - verdverm 3y ago
      nerdctl, colima, containerd
ElectricalUnion 3y ago
> it looks to have a lot of overlap with containers No it doesn't. They're at different abstraction layers. Containers are more convenient form of generic code/data packaging and isolation primitives. llamafile is the code/data that you want to run. Equivalent poor analogy: Why does JPEG XL exists? I can just put a base64 representation of a PNG image in a data URL inside a HTML file inside a ZIP file; I can also bundle more stuff this way! JPEG XL is useless! No one should use JPEG XL! Other use cases that use JPEG XL by themselves are invalid!