3 ms·

For one Ollama supports interleaved sliding window attention for Gemma 3 while llama.cpp doesn't.[0] iSWA reduces kv cache size to 1/6. Ollama is written in go

For one Ollama supports interleaved sliding window attention for Gemma 3 while llama.cpp doesn't.[0] iSWA reduces kv cache size to 1/6. Ollama is written in golang so of course they can not meaningfully contribute that back to llama.cpp. [0] https://github.com/ggml-org/llama.cpp/issues/12637 https://github.com/ggml-org/llama.cpp/issues/12637

noodletheworld 1y ago
What nonsense is this? Where do you imagine ggml is from? > The llama.cpp project is the main playground for developing new features for the ggml library -> https://github.com/ollama/ollama/tree/27da2cddc514208f4e23539e00485b880e3e2191/ml/backend/ggml/ggml https://github.com/ollama/ollama/tree/27da2cddc514208f4e2353... (Hint: If you think they only write go in ollama, look at the commit history of that folder)
- nolist_policy 1y ago
  llama.cpp clearly does not support iSWA: https://github.com/ggml-org/llama.cpp/issues/12637 https://github.com/ggml-org/llama.cpp/issues/12637 Ollama does, please try it.
- imtringued 1y ago
  Dude, they literally announced that they stopped using llama.cpp and are now using ggml directly. Whatever gotcha you think there is, exists only in your head.
  - noodletheworld 1y ago
    I'm responding to this assertion: > Ollama is written in golang so of course they can not meaningfully contribute that back to llama.cpp. llama.cpp consumes GGML. ollama consumes GGML. If they contribute upstream changes, they are contributing to llama.cpp. The assertions that they: a) only write golang b) cannot upstream changes Are both, categorically, false. You can argue what 'meaningfully' means if you like. You can also believe whatever you like. However, both (a) and (b), are false. It is not a matter of dispute. > Whatever gotcha you think there is, exists only in your head. There is no 'gotcha'. You're projecting. My only point is that any claim that they are somehow not able to contribute upstream changes only indicates a lack of desire or competence, not a lack of the technical capacity to do so.
    - refulgentis 1y ago
      FWIW I don't know why you're being downvoted other than a standard from the bleachers "idk what's going on but this guy seems more negative!" -- cheers -- "a [specious argument that shades rather than illuminates] can travel halfway around the world before..."
refulgentis 1y ago
It's impossible to meaningfully contribute to the C library you call from Go because you're calling it from Go? :) We can see the weakness of this argument given it is unlikely any front-end is written in C, and then noting it is unlikely ~0 people contribute to llama.cpp.
- magicalhippo 1y ago
  They can of course meaningfully contribute new C++ code to llama.cpp, which they then could later use downstream in Go. What they cannot meaningfully do is write Go code that solves their problems and upstream those changes to llama.cpp. The former requires they are comfortable writing C++, something perhaps not all Go devs are.
  - refulgentis 1y ago
    I'd love to be able to take this into account, step back, and say "Ah yes - there is non-zero probability they are materially incapable of contributing back to their dependency" - in practice, if you're comfortable writing SWA in Go, you're going to be comfortable writing it in C++, and they are writing C++ already. (it's also worth looking at the code linked for the model-specific impls, this isn't exactly 1000s of lines of complicated code. To wit, while they're working with Georgi...why not offer to help land it in llama.cpp?)
    - magicalhippo 1y ago
      Perhaps for SWA. For the multimodal stuff it's a lot clear cut. Ollama used the image processing libraries from Go, while in llama.cpp they ended up rolling their own image processing routines.
      - refulgentis 1y ago
        Citation? My groundbreaking implementation passes it RGB bytes, passes em through the image projector, and put the tokens in the prompt. And I cannot imagine sure why the inference engine would be more concerned with it than that. Is my implementation a groundbreaking achievement worth rendering llama.cpp a footnote, because I use Dart image-processing libraries?

noodletheworld 1y ago

What nonsense is this? Where do you imagine ggml is from? > The llama.cpp project is the main playground for developing new features for the ggml library -> https://github.com/ollama/ollama/tree/27da2cddc514208f4e23539e00485b880e3e2191/ml/backend/ggml/ggml https://github.com/ollama/ollama/tree/27da2cddc514208f4e2353... (Hint: If you think they only write go in ollama, look at the commit history of that folder)

nolist_policy 1y ago
llama.cpp clearly does not support iSWA: https://github.com/ggml-org/llama.cpp/issues/12637 https://github.com/ggml-org/llama.cpp/issues/12637 Ollama does, please try it.
imtringued 1y ago
Dude, they literally announced that they stopped using llama.cpp and are now using ggml directly. Whatever gotcha you think there is, exists only in your head.
- noodletheworld 1y ago
  I'm responding to this assertion: > Ollama is written in golang so of course they can not meaningfully contribute that back to llama.cpp. llama.cpp consumes GGML. ollama consumes GGML. If they contribute upstream changes, they are contributing to llama.cpp. The assertions that they: a) only write golang b) cannot upstream changes Are both, categorically, false. You can argue what 'meaningfully' means if you like. You can also believe whatever you like. However, both (a) and (b), are false. It is not a matter of dispute. > Whatever gotcha you think there is, exists only in your head. There is no 'gotcha'. You're projecting. My only point is that any claim that they are somehow not able to contribute upstream changes only indicates a lack of desire or competence, not a lack of the technical capacity to do so.
  - refulgentis 1y ago
    FWIW I don't know why you're being downvoted other than a standard from the bleachers "idk what's going on but this guy seems more negative!" -- cheers -- "a [specious argument that shades rather than illuminates] can travel halfway around the world before..."

refulgentis 1y ago

It's impossible to meaningfully contribute to the C library you call from Go because you're calling it from Go? :) We can see the weakness of this argument given it is unlikely any front-end is written in C, and then noting it is unlikely ~0 people contribute to llama.cpp.

magicalhippo 1y ago
They can of course meaningfully contribute new C++ code to llama.cpp, which they then could later use downstream in Go. What they cannot meaningfully do is write Go code that solves their problems and upstream those changes to llama.cpp. The former requires they are comfortable writing C++, something perhaps not all Go devs are.
- refulgentis 1y ago
  I'd love to be able to take this into account, step back, and say "Ah yes - there is non-zero probability they are materially incapable of contributing back to their dependency" - in practice, if you're comfortable writing SWA in Go, you're going to be comfortable writing it in C++, and they are writing C++ already. (it's also worth looking at the code linked for the model-specific impls, this isn't exactly 1000s of lines of complicated code. To wit, while they're working with Georgi...why not offer to help land it in llama.cpp?)
  - magicalhippo 1y ago
    Perhaps for SWA. For the multimodal stuff it's a lot clear cut. Ollama used the image processing libraries from Go, while in llama.cpp they ended up rolling their own image processing routines.
    - refulgentis 1y ago
      Citation? My groundbreaking implementation passes it RGB bytes, passes em through the image projector, and put the tokens in the prompt. And I cannot imagine sure why the inference engine would be more concerned with it than that. Is my implementation a groundbreaking achievement worth rendering llama.cpp a footnote, because I use Dart image-processing libraries?