3 ms·

Ollama's approach with Docker is indeed user-friendly, but Rust + WebAssembly (Wasm) offers some cool perks that are worth considering. With Wasm, you get true

Ollama's approach with Docker is indeed user-friendly, but Rust + WebAssembly (Wasm) offers some cool perks that are worth considering. With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed. It's also super lightweight, making it faster and more resource-efficient – a big plus for running LLMs, especially on devices with limited resources. A Wasm inference application can be as small as 2MB, including all dependencies, which is significantly smaller than most Docker containers.

hobofan 3y ago
Ollama doesn't use Docker by default. While it is one of the install/download options, the default mode of operation is installing it as a native application (where they provide prebuilt binaries for the main platforms/architectures) into which you can easily load models from HuggingFace repos or local GGUF files.
- hiatus 3y ago
  I don't understand. When you run `ollama pull` (or even `ollama run` which will pull if you don't have the model), that is using docker is it not? The code even references pulling from a registry. https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L959 https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L95...
  - _ea1k 3y ago
    ollama maintains a registry of ollama compatible llms. I don't see any indication that they use docker for that registry, but I could have overlooked something. I also don't see any docker/podman containers running when using ollama.
  - hobofan 3y ago
    I haven't looked to deeply into the code, but it looks to me like it just uses OCI/Docker images as a distribution format. You can find some evidence of that in the model directory (mentioned in the FAQ[0]), where you can find manifest files that have contents of a docker manifest. Using this format likely makes distribution a lot easier as they can just use an existing OCI registry for hosting the images, and you can use existing push/pull primitives. OCI images are often used nowadays for things besides docker images if you have a format (like a layered filesystem) that is nicely content-addressable. The default registry for Ollama is https://ollama.ai/library https://ollama.ai/library (I don't know about the API endpoint). I'm not sure if/how you can actually use it with another registry endpoint. When people say that something "uses Docker" that usually involves utilizing the Docker daemon with it's sandboxing, networking, etc.. Ollama doesn't do or need to do that as it can just do inference natively based on GGUF files (+ the other configuration that comes as part of their model file). However, as you have noticed, it does borrow heavily from Docker on a conceptual level. E.g. apart from the registry mechanism its Modelfile heavily mimics a Dockerfile. [0]: https://github.com/jmorganca/ollama/blob/aabd71aede99443d5858213eec4e8d4c10713c55/docs/faq.md#where-are-models-stored https://github.com/jmorganca/ollama/blob/aabd71aede99443d585...
cherryteastain 3y ago
[dead]
LoganDark 3y ago
> With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed. No... no, that is not what Wasm means at all.
deleted 3y ago
[deleted]
Kubuxu 3y ago
> you get true cross-platform support wasi_nn is just an abstraction layers for wrappers around llama.cpp/gguf.cpp/onyx/others. It is not any more cross-platform than these libraries/apps deployed themselves, or rust apps using bindings for them.

hobofan 3y ago

Ollama doesn't use Docker by default. While it is one of the install/download options, the default mode of operation is installing it as a native application (where they provide prebuilt binaries for the main platforms/architectures) into which you can easily load models from HuggingFace repos or local GGUF files.

hiatus 3y ago
I don't understand. When you run `ollama pull` (or even `ollama run` which will pull if you don't have the model), that is using docker is it not? The code even references pulling from a registry. https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L959 https://github.com/jmorganca/ollama/blob/main/cmd/cmd.go#L95...
- _ea1k 3y ago
  ollama maintains a registry of ollama compatible llms. I don't see any indication that they use docker for that registry, but I could have overlooked something. I also don't see any docker/podman containers running when using ollama.
- hobofan 3y ago
  I haven't looked to deeply into the code, but it looks to me like it just uses OCI/Docker images as a distribution format. You can find some evidence of that in the model directory (mentioned in the FAQ[0]), where you can find manifest files that have contents of a docker manifest. Using this format likely makes distribution a lot easier as they can just use an existing OCI registry for hosting the images, and you can use existing push/pull primitives. OCI images are often used nowadays for things besides docker images if you have a format (like a layered filesystem) that is nicely content-addressable. The default registry for Ollama is https://ollama.ai/library https://ollama.ai/library (I don't know about the API endpoint). I'm not sure if/how you can actually use it with another registry endpoint. When people say that something "uses Docker" that usually involves utilizing the Docker daemon with it's sandboxing, networking, etc.. Ollama doesn't do or need to do that as it can just do inference natively based on GGUF files (+ the other configuration that comes as part of their model file). However, as you have noticed, it does borrow heavily from Docker on a conceptual level. E.g. apart from the registry mechanism its Modelfile heavily mimics a Dockerfile. [0]: https://github.com/jmorganca/ollama/blob/aabd71aede99443d5858213eec4e8d4c10713c55/docs/faq.md#where-are-models-stored https://github.com/jmorganca/ollama/blob/aabd71aede99443d585...

cherryteastain 3y ago

[dead]

LoganDark 3y ago

> With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automagically uses the local hardware accelerator to run at full speed. No... no, that is not what Wasm means at all.

deleted 3y ago

[deleted]

Kubuxu 3y ago

> you get true cross-platform support wasi_nn is just an abstraction layers for wrappers around llama.cpp/gguf.cpp/onyx/others. It is not any more cross-platform than these libraries/apps deployed themselves, or rust apps using bindings for them.