6 ms·
Ollama's new engine for multimodal models
- deleted 1y ago[deleted]
- ics 1y agoI'll have to try this later but appreciate that the article gets straight to the point with practical examples and then the details.
- newusertoday 1y agowhy does ollama engine has to change to support new models? every time a new model comes ollama has to be upgraded.
- nkwaml 1y agoBecause of things like this: https://github.com/ggml-org/llama.cpp/issues/12637 https://github.com/ggml-org/llama.cpp/issues/12637 Where "supporting" a model doesn't mean what you think it means for cpp Between that and the long saga with vision models having only partial support, with a CLI tool, and no llama-server support (they only fixed all that very recently) the fact of the matter is that ollama is moving faster and implementing what people want before lama.cpp now And it will finally shut down all the people who kept copy pasting the same criticism of ollama "it's just a llama.cpp wrapper why are you not using cpp instead"
- Maxious 1y agoThere's also some interpersonal conflict in llama.cpp that's hampering other bug fixes https://github.com/ikawrakow/ik_llama.cpp/pull/400 https://github.com/ikawrakow/ik_llama.cpp/pull/400
- kergonath 1y agoWhat the hell is going on there? It’s utterly bizarre to see devs discussing granting each other licences to work on the same code for an open source project. How on earth did they end up there?
- Philpax 1y agoThere seems to be some bad blood between ikawrakow and ggerganov: https://github.com/ikawrakow/ik_llama.cpp/discussions/316 https://github.com/ikawrakow/ik_llama.cpp/discussions/316
- tough 1y agoBut he's talking about a MIT License! WTF
- prophesi 1y agoMy guess is that there's money involved. Maybe a spat between an ex-employee and their ex-employer?
- w8nC 1y agoNow it’s just a wrapper around hosted APIs. Went with my own wrapper around llama.cpp and stable-diffusion.cpp with optional prompting hosted if I don’t like the result so much, but it makes a good start for hosted to improve on. Also obfuscates any requests sent to hosted, cause why feed them insight to my use case when I just want to double check algorithmic choices of local AI? The ground truth relationship func names and variable names imply is my little secret
- Patrick_Devine 1y agoWait, what hosted APIs is Ollama wrapping?
- simonw 1y agoThe timing on this is a little surprising given llama.cpp just finally got a (hopefully) stable vision feature merged into main: https://simonwillison.net/2025/May/10/llama-cpp-vision/ https://simonwillison.net/2025/May/10/llama-cpp-vision/ Presumably Ollama had been working on this for quite a while already - it sounds like they've broken their initial dependency on llama.cpp. Being in charge of their own destiny makes a lot of sense.
- lolinder 1y agoDo you know what exactly the difference is with either of these projects adding multimodal support? Both have supported LLaVA for a long time. Did that require special casing that is no longer required? I'd hoped to see this mentioned in TFA, but it kind of acts like multimodal is totally new to Ollama, which it isn't.
- simonw 1y agoThere's a pretty clear explanation of the llama.cpp history here: https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd#multimodal-support-in-llamacpp https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd... I don't fully understand Ollama's timeline and strategy yet.
- refulgentis 1y agoIt's a turducken of crap from everyone but ngxson and Hugging Face and llama.cpp in this situation. llama.cpp did have multimodal, I've been maintaining an integration for many moons now. (Feb 2024? Original LLaVa through Gemma 3) However, this was not for mere mortals. It was not documented and had gotten unwieldy, to say the least. ngxson (HF employee) did a ton of work to get gemma3 support in, and had to do it in a separate binary. They dove in and landed a refactored backbone that is presumably more maintainable and on track to be in what I think of as the real Ollama, llama.cpp's server binary. As you well note, Ollama is Ollamaing - I joked, once, that the median llama.cpp contribution from Ollama is a driveby GitHub comment asking when a feature will land in llama-server, so it can be copy-pasted into Ollama. It's really sort of depressing to me because I'm just one dude, it really wasn't that hard to support it (it's one of a gajillion things I have to do, I'd estimate 2 SWE-weeks at 10 YOE, 1.5 SWE-days for every model release), and it's hard to get attention for detailed work in this space with how much everyone exaggerates and rushes to PR. EDIT: Coming back after reading the blog post, and I'm 10x as frustrated. "Support thinking / reasoning; Tool calling with streaming responses" --- this is table stakes stuff that was possible eons ago. I don't see any sign of them doing anything specific in any of the code they link, the whole thing reads like someone carefully worked with an LLM to present a maximalist technical-sounding version of the llama.cpp stuff and frame it as if they worked with these companies and built their own thing. (note the very careful wording on this, e.g. in the footer the companies are thanked for releasing the models) I think it's great that they have a nice UX that helps people run llama.cpp locally without compiling, but it's hard for me to think of a project I've been more by turned off by in my 37 years on this rock.
- tommica 1y agoSidetangent: why is ollama frowned upon by some people? I've never really got any other explanation than "you should run llama.CPP yourself"
- nicman23 1y agocpp was just faster and with more features that is all
- cwillu 1y agocpp is the thing doing all the heavy lifting, ollama is just a library wrapper. It'd be like if handbrake tried to pretend that they implemented all the video processing work, when it's dependent on libffmpeg for all of that.
- diggan 1y ago> ollama is just a library wrapper. Was. This submission is literally about them moving away from being just a wrapper around llama.cpp :)
- buyucu 1y agono they are not. the submission uses ggml, which is llama.cpp
- diggan 1y agoI think you misunderstand how these pieces fit together. llama.cpp is library that ships with a CLI+some other stuff, ggml is a library and Ollama has "runners" (like an "execution engine"). Previously, Ollama used llama.cpp (which uses ggml) as the only runner. Eventually, Ollama made their own runner (which also uses ggml) for new models (starting with gemma3 maybe?), still using llama.cpp for the rest (last time I checked at least). ggml != llama.cpp, but llama.cpp and Ollama are both using ggml as a library.
- buyucu 1y ago[flagged]
- hexmiles 1y agoIf I understood it correctly: this time no, it is actually new engine builded by the ollama team indipendent from llama.cpp
- Havoc 1y agollama.cpp added support for vision 6 days ago. See SimonW post here: https://simonwillison.net/2025/May/10/llama-cpp-vision/ https://simonwillison.net/2025/May/10/llama-cpp-vision/ >If I understood it correctly You understood it exactly like they wanted you to...
- buyucu 1y agoI doubt it. Llama.cpp just added support for the same models a few weeks ago. Folks at ollama just did a git pull.
- magicalhippo 1y agoIt's open source, you could have checked. Seems indeed like the new engine cuts out llama.cpp, using GGML libary directly. https://github.com/ollama/ollama/pull/7913 https://github.com/ollama/ollama/pull/7913
- buyucu 1y agoseriously? who do you think develops ggml? hint: it's llama.cpp
- deleted 1y ago[deleted]
- oezi 1y agoI wish multimodal would imply text, image and audio (+potentially video). If a model supports only image generation or image analysis, vision model seems the more appropriate term. We should aim to distinguish multimodal modals such as Qwen2.5-Omni from Qwen2.5-VL. In this sense: Ollama's new engine adds vision support.
- deleted 1y ago[deleted]
- prettyblocks 1y agoI'm very interested in working with video inputs, is it possible to do that with Qwen2.5-Omni and Ollama?
- oezi 1y agoI have only tested Qwen2.5-Omni for audio and it was hit and miss for my use case of tagging audio.
- machinelearning 1y agoWhat's a use case are you interested in re: video?
- prettyblocks 1y agoI'm curious how effective these models would be at recognizing if the input video was ai generated or heavily manipulated. Also various things around face/object segmentation.
- tough 1y agohttps://huggingface.co/blog/smolvlm https://huggingface.co/blog/smolvlm
- Koshima 1y agoThe timing makes sense if you consider the broader trend in the LLM space. We're moving from just text to more integrated, multimodal experiences, and having a tightly controlled engine like this could be a game changer for developers building apps that require real-time, context-rich understanding.
- bradly 1y agoThe strength with Ollama for me was the ease of being able to run a simple Docker command and be up and running locally without any tinkering, but with image and video Docker is no longer an option as Docker does not use the GPU. I'm curious how Ollama plans to support their Docker integration going forward or if it is a less important part of the project that I'm giving it credit for.
- IanCal 1y agoYou can use a GPU with docker - at least on some platforms. There's more setup though, nvidia have some details to help https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html https://docs.nvidia.com/datacenter/cloud-native/container-to...
- bradly 1y agoThank you. I should have specified on MacOS. I ran into this recently trying to setup stable-diffusion-webui/InvokeAI/Foocus and finding it much more complicated to get working for me on my personal laptop than the llms.
- washadjeffmad 1y agoOut of curiosity, before you attempted this, what was your impression of the fitness and performance of Macs for generative AI?
- bradly 1y agoBefore I attempted, I had no idea. I hadn't ran any AI models locally and I don't follow this stuff too closely, so I wasn't even sure if I could get something usable on my M1 MacBook Air. I went in fairly blind which is why the Ollama Docker installer was so appealing to me–I got to hold off fighting Python and Homebrew until I had a better sense of what the tool could provide. After my attempt, I think chat is performant enough on my M1. Code gen was too slow for me. Image generation was 1-2 minutes for small pixel art sprites, which for my use case is fine to let churn for a while, but the image generation results were much worse than ChatGPT browser gives me out of the box. I do not know if poor image quality is due to machine constraints or me not understanding how to configure the checkpoint and models. I would be interested to hear how an M3 or M4 Mini handles these things as those are fair affordable to pick up used.
- mark_l_watson 1y agoI have mostly used Ollama to run local models for close to a year, love it, but I have barely touched Llava, etc. multi modal support because all my personal use cases are text based. Question: what are cool and useful multi modal projects have people here built using local models? I am looking for personal project ideas.
- JKCalhoun 1y agoDoes Ollama support the "user context" that higher level LLMs like ChatGPT have? I'm not clear what they are called (or how implemented) — but perhaps 1) the initial prompt/context (that, for example, Grok has got in trouble with recently) and 2) the kind of saved context that allows ChatGPT to know things about your prompt-history so it can better answer future queries. (My use of ollama has been pretty bare-bones and I have not seen anything covering these higher level features in -help.)
- lxgr 1y agoMy understanding is that ollama is more of an "LLM backend", i.e. it provides a server process on your machine that answers requests relatively statelessly. I believe it keeps the model loaded across sessions, and possibly keeps the KV cache warm for ongoing sessions (but I doubt it, based on the API shape; I don't see a "session" parameter), but that's about it. Nothing seems to be written to disk. Features like ChatGPT's "memories" or cross-chat context require a persistence layer that's probably best suited for a "frontend". Ollama's API does support passing in requests with history, for example: https://github.com/ollama/ollama/blob/main/docs/api.md#chat-request-with-history https://github.com/ollama/ollama/blob/main/docs/api.md#chat-...
- codybontecou 1y agoIs there more to memory than just an entry into the context/messages array passed to the LLM?
- lxgr 1y agoThere must be some heavy compression/filtering going on, as there's no chance GPT can hold everybody's entire ChatGPT conversation history in its context. But practically, I believe that Ollama just doesn't have a concept of server-side persistent state at the moment to even do such a thing.
- codybontecou 1y ago
- yossi_peti 1y agoTheir example "understanding and translating vertical Chinese spring couplets to English" has a lot of mistakes in it. I'm guessing the person writing the blog post to show off that example doesn't actually know Chinese. What is actually written: Top: 家和国盛 Left: 和谐生活人人舒畅迎新春 Right: 平安社会家家欢乐辞旧岁 What Ollama saw: Top: 盛和家国 (correct characters but wrong order) Left: It reads "新春" (new spring) as 舒畅 (comfortable) Right: 家家欢欢乐乐辞旧岁 (duplicates characters and omits the first four)
- mchiang 1y agoI'm one of the maintainers who ran that example. I am Chinese. The English translation, I thought was pretty spot on. We don't hide the mistakes of the models or fake the demos. Overtime, of course I hope the models to improve much more
- clpm4j 1y agoThe whole '*llama' naming convention in the LLM world is more confusing to me than it probably should be. So many llamas running around out here.
- mcbuilder 1y agoUnfortunately the speed of AI/ML is so crazy fast. I don't know a better way to keep track other than paying attention all the time. The field also loves memey names. A few years ago everyone was naming models after Sesame Street characters, there were the YOLO family of models. Conference papers are not immune, in fact they are greatest "offenders".
- deleted 1y ago[deleted]
- andy_xor_andrew 1y agoThey are talking a lot about this new engine - I'd love to see details on how it's actually implemented. Given llama.cpp is a herculean feat, if you are going to claim to have some replacement for it, an example of how you did it would be good! Based on this part: > We set out to support a new engine that makes multimodal models first-class citizens, and getting Ollama’s partners to contribute more directly the community - the GGML tensor library. And from clicking through a github link they had: https://github.com/ollama/ollama/blob/main/model/models/gemma3/model_text.go https://github.com/ollama/ollama/blob/main/model/models/gemm... My takeaway is, the GGML library (the thing that is the backbone for llama.cpp) must expose some FFI (foreign function interface) that can be invoked from Go, so in the ollama Go code, they can write their own implementations of model behavior (like Gemma 3) that just calls into the GGML magic. I think I have that right? I would have expected a detail like that to be front and center in the blog post.
- Hugsun 1y agoOllama are known for their lack of transparency, poor attribution and anti-user decisions. I was surprised to see the amount of attribution in this post. They've been catching quite a bit of flack for this so they might be adjusting.
- ac29 1y agoI am amused that one of the handful examples they chose to use is wrong: "The best way to get to Stanford University from the Ferry Building in San Francisco depends on your preferences and budget. Here are a few options: 1. *By Car*: Take US-101 South to CA-85 South, then continue on CA-101 South." CA 85 is significantly farther down 101 than Palo Alto.