5 ms·

Ollama and gguf

173 points by indigodaddy 1y ago

indigodaddy 1y ago
ggerganov explains the issue: https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...
- magicalhippo 1y ago
  I noticed it the other way, llama.cpp failed to download the Ollama-downloaded gpt-oss 20b model. Thought it was odd given all the others I tried worked fine. Figured it had to be Ollama doing Ollama things, seems that was indeed the case.
  - magicalhippo 1y ago
    Oh wow... > llama.cpp failed to download the Ollama-downloaded gpt-oss 20b model That should of course read "llama.cpp failed to load the Ollama-downloaded gpt-oss 20b model"
- polotics 1y ago
  ggerganov is my hero, and... it's a good thing this got posted so I saw in the comments that --flash-attn --cache-reuse 256 could help with my setup (M3 36GB + RPC to M1 16GB) figuring out what params to set and at what value is a lot of trial and error, Gemini does help a bit clarify what params like top-k are going to do in practice. Still the whole load-balancing with RPC is something I think I'm going to have to read the source of llama.cpp to really understand (oops I almost wrote grok, damn you Elon) Anyways ollama is still not doing distributed load, and yeah I guess using it is a stepping stone...
- LeoPanthera 1y ago
  The named anchor in this URL doesn't work in Safari. Safari correctly scrolls down to the comment in question, but then some Javascript on the page throws you back up to the top again.
- scosman 1y ago
  This is the comment people should read. GG is amazing. Ollama forked to get it working for day 1 compatibility. They need to get their system back in line with mainline because of that choice. That's kinda how open source works. The uproar over this (mostly on reddit and x) seems unwarranted. New models regularly have compatibility issues for much longer than this.
  - ekianjo 1y ago
    GG clearly mentioned they did not contribute anything to upstream.
- buyucu 1y ago
  ggerganov is a treasure. the man deserves a medal.
dcreater 1y ago
I think the title buries the lede? Its specific to GPT-OSS and exposes the shady stuff Ollama is doing to acquiesce/curry favor/partner with/get paid by corporate interests
- freedomben 1y ago
  I think "shady" is a little too harsh - sounds like they forked an important upstream project, made incompatible changes that they didn't push upstream or even communicate with upstream about, and now have to deal with the consequences of that. If that's "shady" (despite being all out in the open) then nearly every company I've worked for has been "shady."
  - wsgeorge 1y ago
    There's a reddit thread from a few months ago that sort of explains what people don't like about ollama, that "shadiness" parent references: https://www.reddit.com/r/LocalLLaMA/comments/1jzocoo/finally_someone_noticed_this_unfair_situation/ https://www.reddit.com/r/LocalLLaMA/comments/1jzocoo/finally...
llmthrowaway 1y ago
Confusing title - thought this was about Ollama finally supporting sharded GGUF (ie. the Huggingface default for large gguf over 48gb). https://github.com/ollama/ollama/issues/5245 https://github.com/ollama/ollama/issues/5245 Sadly it is not and the issue still remains open after over a year meaning ollama cannot run the latest SOTA open source models unless they covert them to their proprietary format which they do not consistently do. No surprise I guess given they've taken VC money, refuse to properly attribute the use things like llama.cpp and ggml, have their own model format for.. reasons? and have over 1800 open issues... Llama-server, ramallama or whatever model switcher ggerganov is working on (he showed previews recently) feel like the way forward.
- brabel 1y ago
  I want to add an inference engine to my product. I was hoping to use ollama because it really helps, I think, make sure you have a model with the right metadata that you can count on working (I've seen that with llama.cpp, it's easy to get the metadata wrong and start getting rubbish from the LLM because the "stop_token" was wrong or something). I'd thought ollama was a proponent of the GGUF, which I really like as it standardizes metadata?! What would be the best way to use llama.cpp and models that use GGUF these days? ramallama is a good alternative (I guess it is, but it's not completely clear from your message)? Or just use llama.cpp directly, in which case how to ensure I don't get rubbish (like the model asking and answering questions by itself without ever stopping)??
  - llmthrowaway 1y ago
    Meant to say llama-swap instead of llama-server. llama-swap adds a gui and dynamic model switching on top of llama-server. Somewhat tricky to set up as it relies on a .yaml file that is poorly documented for using with docker but something like: "GLM4-Air": env: - "CUDA_VISIBLE_DEVICES=1" proxy: "http://127.0.0.1:9999" cmd: > /app/llama-server --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn --ctx-size 32684 --jinja -ngl 20 --model /modelfiles/GLM-4.5-Air-UD-Q4_K_XL-00001-of-00002.gguf --port 9999 When run via docker this gets you a similar setup to ollama. The yaml file also needs TTL set if you want it to unload models after an idle period. Ollama native models in their marketplace have these params supposedly set correctly to save you having to do this config but in practice this is hit or miss and often these change from day 0 of the release.
tarruda 1y ago
I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess listening on a unix socket, and forward requests to it. One thing I'm curious about: Does ollama support strict structured output or strict tool calls adhering to a json schema? Because it would be insane to rely on a server for agentic use unless your server can guarantee the model will only produce valid json. AFAIK this feature is implemented by llama.cpp, which they no longer use.
- arcanemachiner 1y ago
  > I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Here is some relevant drama on the subject: https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...
- hodgehog11 1y ago
  I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contracts with simultaneous releases if there is no guarantee the model will even function. By reimplementing this layer, Ollama gets to enjoy a kind of LTS status that their partners rely on. It won't be as feature-rich, and definitely won't be as fast, but that's not their goal.
  - A4ET8a8uTh0_v2 1y ago
    Thank you. This is genuinely a valid reason even from a simple consistency perspective. (edit: I think -- after I read some of the links -- I understand why Ollama comes across as less of a hero. Still, I am giving them some benefit of the doubt since they made local models very accessible to plebs like me; and maybe I can graduate to no ollama )
12345hn6789 1y ago
Just days ago ollama devs claimed[0] that ollama no longer relies on ggml / llama.cpp. here is their pull request(+165,966 −47,980) to reimplement (copy) llama.cpp code in their repository. https://github.com/ollama/ollama/pull/11823 https://github.com/ollama/ollama/pull/11823 [0] https://news.ycombinator.com/item?id=44802414#44805396 https://news.ycombinator.com/item?id=44802414#44805396
- flakiness 1y ago
  not against overall sentiment here, but quote the counterpoint from the linked HN comment to be fair: > Ollama does not use llama.cpp anymore; we do still keep it and occasionally update it to remain compatible for older models for when we used it. The linked PR is doing "occasionally update it" I guess? Note that "vendored" in the PR title often means to take a snapshot to pin a specific version.
  - ekianjo 1y ago
    gpt-oss is not an "older model"
- gcr 1y ago
  The PR you linked to says “thanks to the amazing work done by ggml-org” and doesn’t remove GGML code, it instead updates the vendored version and seems to throw away ollama’s custom changes. That’s the opposite of disentangling. Here’s the maintainer of ggml explaining the context behind this change: https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...
om8 1y ago
llama.cpp is a mess and ollama is right to move on from it
- buyucu 1y ago
  ollama is still using llama.cpp. they are just denying that they are :)
iamshrimpy 1y ago
This title makes no sense and it links nowhere helpful. It's "Ollama's forked ggml is incompatible with other gpt-oss GGUFs" and it should link to GG's comment[0] [0] https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...
- indigodaddy 1y ago
  HN strips any "?|#" etc at the end of URLs, and you cannot edit the URL after submissions (like you can the title). As far as the title, yeah I didn't work too hard on making it good, sorry
am17an 1y ago
There’s a GitHub link which is open from last year, about the missing license in ollama. They have not bothered to reply, which goes to show how much they care. Also it’s a YC company, I see more and more morally bankrupt companies making the cut recently, why is that?
- pduggishetti 1y ago
  I think most of them were morally bankrupt, you might just be realizing now.
sunnycoder5 1y ago
for folks wrestling with Ollama, llama.cpp or local LLM versioning - did you guys check out Docker's new feature - Docker Model Runner? Docker Model Runner makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub or any OCI-compliant registry. Whether you're building generative AI applications, experimenting with machine learning workflows, or integrating AI into your software development lifecycle, Docker Model Runner provides a consistent, secure, and efficient way to work with AI models locally. For more details check this out : https://docs.docker.com/ai/model-runner/ https://docs.docker.com/ai/model-runner/
- BretFisher 1y ago
  I have a video on using DMR https://youtu.be/3p2uWjFyI1U?si=dazN8rRRdIbAa8Sl https://youtu.be/3p2uWjFyI1U?si=dazN8rRRdIbAa8Sl
buyucu 1y ago
ollama is a lost cause. they are going through a very aggressive phase of enshittification right now.
- elaus 1y ago
  What is the currently favored alternative for simply running 1-2 models locally, exposed via an API? One big advantage of Ollama seems to be that they provide fully configured models, so I don't have to fiddle with stop words, etc.
  - buyucu 1y ago
    I just use llama.cpp server. It works really well. Some people recommend llama-swap or kobold but I never tried them.
  - arcanemachiner 1y ago
    llama-swap if you need more than 1 model. It wraps llama.cpp, and has a Docker container version that is pretty easy to work with.
- mark_l_watson 1y ago
  I disagree. Ollama’s reason to be is to make things simple, not to always be on the cutting edge. I use Ollama when I can because of this simplicity. Since I bought a 32G integrated memory Mac 18 months ago, I have run so many models using Ollama, with close to zero problems. The simple thing to do is to just use the custom quantization that OpenAI used for gpt-oss and use GGUF for other models. Using Huggingface, LM Studio, etc. is the Linux metaphor of flexibility. Using Ollama is sort of like using macOS
deleted 1y ago
[deleted]
diimdeep 1y ago
ollama is certain in the future rent seeking wrapper by docker fame. classic Docker Hub playbook: spread the habit for free → capture workflows → charge for scale. moat isn't in inference speed — it’s in controlling the distribution & default UX, once they own that, can start rent-gate it.
clarionbell 1y ago
Why is anyone still using this? You can spin up llama.cpp server and have more optimized runtime. And if you insist on containers you can go for ramallama https://ramalama.ai/ https://ramalama.ai/
- buyucu 1y ago
  I think people just don't know any better. I also used Ollama way longer than I should have. I didn't know that Ollama was just llama.cpp with a thin wrapper. My quality of life improved a lot after I discovered llama.cpp.
- jjice 1y ago
  Ollama is a more out of the box solution. I also prefer llama.cpp for the more FOSS aspects, but Ollama is a simpler install, model download (this is the biggest convenience IMO), and execution. For those reasons, that's why I believe it's still fairly popular as a solution.
  - dd3boh 1y ago
    By the way, you can download models straight from hugging face with llama.cpp. It might be a few characters longer than the command you would run on ollama, but still.
    - brabel 1y ago
      Then you need to also provide appropriate metadata and format messages correctly according to the format. Which I believe llama.cpp doesn’t do by default, or it can do it? I had trouble formatting messages correctly using llama.cpp due to possibly mismatch in metadata, which ollama seems to handle, but would love to know if this is wrong.
      - dpkirchner 1y ago
        Plus a huggingface token to access models that require you to beg for approval. Ollama hosted models don't require that (which may not be legit but most users don't care).

173 points by indigodaddy 1y ago

ggerganov explains the issue: https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...

magicalhippo 1y ago
I noticed it the other way, llama.cpp failed to download the Ollama-downloaded gpt-oss 20b model. Thought it was odd given all the others I tried worked fine. Figured it had to be Ollama doing Ollama things, seems that was indeed the case.
- magicalhippo 1y ago
  Oh wow... > llama.cpp failed to download the Ollama-downloaded gpt-oss 20b model That should of course read "llama.cpp failed to load the Ollama-downloaded gpt-oss 20b model"
polotics 1y ago
ggerganov is my hero, and... it's a good thing this got posted so I saw in the comments that --flash-attn --cache-reuse 256 could help with my setup (M3 36GB + RPC to M1 16GB) figuring out what params to set and at what value is a lot of trial and error, Gemini does help a bit clarify what params like top-k are going to do in practice. Still the whole load-balancing with RPC is something I think I'm going to have to read the source of llama.cpp to really understand (oops I almost wrote grok, damn you Elon) Anyways ollama is still not doing distributed load, and yeah I guess using it is a stepping stone...
LeoPanthera 1y ago
The named anchor in this URL doesn't work in Safari. Safari correctly scrolls down to the comment in question, but then some Javascript on the page throws you back up to the top again.
scosman 1y ago
This is the comment people should read. GG is amazing. Ollama forked to get it working for day 1 compatibility. They need to get their system back in line with mainline because of that choice. That's kinda how open source works. The uproar over this (mostly on reddit and x) seems unwarranted. New models regularly have compatibility issues for much longer than this.
- ekianjo 1y ago
  GG clearly mentioned they did not contribute anything to upstream.
buyucu 1y ago
ggerganov is a treasure. the man deserves a medal.

dcreater 1y ago

I think the title buries the lede? Its specific to GPT-OSS and exposes the shady stuff Ollama is doing to acquiesce/curry favor/partner with/get paid by corporate interests

freedomben 1y ago
I think "shady" is a little too harsh - sounds like they forked an important upstream project, made incompatible changes that they didn't push upstream or even communicate with upstream about, and now have to deal with the consequences of that. If that's "shady" (despite being all out in the open) then nearly every company I've worked for has been "shady."
- wsgeorge 1y ago
  There's a reddit thread from a few months ago that sort of explains what people don't like about ollama, that "shadiness" parent references: https://www.reddit.com/r/LocalLLaMA/comments/1jzocoo/finally_someone_noticed_this_unfair_situation/ https://www.reddit.com/r/LocalLLaMA/comments/1jzocoo/finally...

llmthrowaway 1y ago

Confusing title - thought this was about Ollama finally supporting sharded GGUF (ie. the Huggingface default for large gguf over 48gb). https://github.com/ollama/ollama/issues/5245 https://github.com/ollama/ollama/issues/5245 Sadly it is not and the issue still remains open after over a year meaning ollama cannot run the latest SOTA open source models unless they covert them to their proprietary format which they do not consistently do. No surprise I guess given they've taken VC money, refuse to properly attribute the use things like llama.cpp and ggml, have their own model format for.. reasons? and have over 1800 open issues... Llama-server, ramallama or whatever model switcher ggerganov is working on (he showed previews recently) feel like the way forward.

brabel 1y ago
I want to add an inference engine to my product. I was hoping to use ollama because it really helps, I think, make sure you have a model with the right metadata that you can count on working (I've seen that with llama.cpp, it's easy to get the metadata wrong and start getting rubbish from the LLM because the "stop_token" was wrong or something). I'd thought ollama was a proponent of the GGUF, which I really like as it standardizes metadata?! What would be the best way to use llama.cpp and models that use GGUF these days? ramallama is a good alternative (I guess it is, but it's not completely clear from your message)? Or just use llama.cpp directly, in which case how to ensure I don't get rubbish (like the model asking and answering questions by itself without ever stopping)??
- llmthrowaway 1y ago
  Meant to say llama-swap instead of llama-server. llama-swap adds a gui and dynamic model switching on top of llama-server. Somewhat tricky to set up as it relies on a .yaml file that is poorly documented for using with docker but something like: "GLM4-Air": env: - "CUDA_VISIBLE_DEVICES=1" proxy: "http://127.0.0.1:9999" cmd: > /app/llama-server --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn --ctx-size 32684 --jinja -ngl 20 --model /modelfiles/GLM-4.5-Air-UD-Q4_K_XL-00001-of-00002.gguf --port 9999 When run via docker this gets you a similar setup to ollama. The yaml file also needs TTL set if you want it to unload models after an idle period. Ollama native models in their marketplace have these params supposedly set correctly to save you having to do this config but in practice this is hit or miss and often these change from day 0 of the release.

tarruda 1y ago

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess listening on a unix socket, and forward requests to it. One thing I'm curious about: Does ollama support strict structured output or strict tool calls adhering to a json schema? Because it would be insane to rely on a server for agentic use unless your server can guarantee the model will only produce valid json. AFAIK this feature is implemented by llama.cpp, which they no longer use.

arcanemachiner 1y ago
> I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Here is some relevant drama on the subject: https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...
hodgehog11 1y ago
I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contracts with simultaneous releases if there is no guarantee the model will even function. By reimplementing this layer, Ollama gets to enjoy a kind of LTS status that their partners rely on. It won't be as feature-rich, and definitely won't be as fast, but that's not their goal.
- A4ET8a8uTh0_v2 1y ago
  Thank you. This is genuinely a valid reason even from a simple consistency perspective. (edit: I think -- after I read some of the links -- I understand why Ollama comes across as less of a hero. Still, I am giving them some benefit of the doubt since they made local models very accessible to plebs like me; and maybe I can graduate to no ollama )

12345hn6789 1y ago

Just days ago ollama devs claimed[0] that ollama no longer relies on ggml / llama.cpp. here is their pull request(+165,966 −47,980) to reimplement (copy) llama.cpp code in their repository. https://github.com/ollama/ollama/pull/11823 https://github.com/ollama/ollama/pull/11823 [0] https://news.ycombinator.com/item?id=44802414#44805396 https://news.ycombinator.com/item?id=44802414#44805396

flakiness 1y ago
not against overall sentiment here, but quote the counterpoint from the linked HN comment to be fair: > Ollama does not use llama.cpp anymore; we do still keep it and occasionally update it to remain compatible for older models for when we used it. The linked PR is doing "occasionally update it" I guess? Note that "vendored" in the PR title often means to take a snapshot to pin a specific version.
- ekianjo 1y ago
  gpt-oss is not an "older model"
gcr 1y ago
The PR you linked to says “thanks to the amazing work done by ggml-org” and doesn’t remove GGML code, it instead updates the vendored version and seems to throw away ollama’s custom changes. That’s the opposite of disentangling. Here’s the maintainer of ggml explaining the context behind this change: https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...

om8 1y ago

llama.cpp is a mess and ollama is right to move on from it

buyucu 1y ago
ollama is still using llama.cpp. they are just denying that they are :)

iamshrimpy 1y ago

This title makes no sense and it links nowhere helpful. It's "Ollama's forked ggml is incompatible with other gpt-oss GGUFs" and it should link to GG's comment[0] [0] https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576 https://github.com/ollama/ollama/issues/11714#issuecomment-3...

indigodaddy 1y ago
HN strips any "?|#" etc at the end of URLs, and you cannot edit the URL after submissions (like you can the title). As far as the title, yeah I didn't work too hard on making it good, sorry

am17an 1y ago

There’s a GitHub link which is open from last year, about the missing license in ollama. They have not bothered to reply, which goes to show how much they care. Also it’s a YC company, I see more and more morally bankrupt companies making the cut recently, why is that?

pduggishetti 1y ago
I think most of them were morally bankrupt, you might just be realizing now.

sunnycoder5 1y ago

for folks wrestling with Ollama, llama.cpp or local LLM versioning - did you guys check out Docker's new feature - Docker Model Runner? Docker Model Runner makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub or any OCI-compliant registry. Whether you're building generative AI applications, experimenting with machine learning workflows, or integrating AI into your software development lifecycle, Docker Model Runner provides a consistent, secure, and efficient way to work with AI models locally. For more details check this out : https://docs.docker.com/ai/model-runner/ https://docs.docker.com/ai/model-runner/

BretFisher 1y ago
I have a video on using DMR https://youtu.be/3p2uWjFyI1U?si=dazN8rRRdIbAa8Sl https://youtu.be/3p2uWjFyI1U?si=dazN8rRRdIbAa8Sl

buyucu 1y ago

ollama is a lost cause. they are going through a very aggressive phase of enshittification right now.

elaus 1y ago
What is the currently favored alternative for simply running 1-2 models locally, exposed via an API? One big advantage of Ollama seems to be that they provide fully configured models, so I don't have to fiddle with stop words, etc.
- buyucu 1y ago
  I just use llama.cpp server. It works really well. Some people recommend llama-swap or kobold but I never tried them.
- arcanemachiner 1y ago
  llama-swap if you need more than 1 model. It wraps llama.cpp, and has a Docker container version that is pretty easy to work with.
mark_l_watson 1y ago
I disagree. Ollama’s reason to be is to make things simple, not to always be on the cutting edge. I use Ollama when I can because of this simplicity. Since I bought a 32G integrated memory Mac 18 months ago, I have run so many models using Ollama, with close to zero problems. The simple thing to do is to just use the custom quantization that OpenAI used for gpt-oss and use GGUF for other models. Using Huggingface, LM Studio, etc. is the Linux metaphor of flexibility. Using Ollama is sort of like using macOS

deleted 1y ago

[deleted]

diimdeep 1y ago

ollama is certain in the future rent seeking wrapper by docker fame. classic Docker Hub playbook: spread the habit for free → capture workflows → charge for scale. moat isn't in inference speed — it’s in controlling the distribution & default UX, once they own that, can start rent-gate it.

clarionbell 1y ago

Why is anyone still using this? You can spin up llama.cpp server and have more optimized runtime. And if you insist on containers you can go for ramallama https://ramalama.ai/ https://ramalama.ai/

buyucu 1y ago
I think people just don't know any better. I also used Ollama way longer than I should have. I didn't know that Ollama was just llama.cpp with a thin wrapper. My quality of life improved a lot after I discovered llama.cpp.
jjice 1y ago
Ollama is a more out of the box solution. I also prefer llama.cpp for the more FOSS aspects, but Ollama is a simpler install, model download (this is the biggest convenience IMO), and execution. For those reasons, that's why I believe it's still fairly popular as a solution.
- dd3boh 1y ago
  By the way, you can download models straight from hugging face with llama.cpp. It might be a few characters longer than the command you would run on ollama, but still.
  - brabel 1y ago
    Then you need to also provide appropriate metadata and format messages correctly according to the format. Which I believe llama.cpp doesn’t do by default, or it can do it? I had trouble formatting messages correctly using llama.cpp due to possibly mismatch in metadata, which ollama seems to handle, but would love to know if this is wrong.
    - dpkirchner 1y ago
      Plus a huggingface token to access models that require you to beg for approval. Ollama hosted models don't require that (which may not be legit but most users don't care).