11 ms·
Ollama now supports AMD graphics cards
- sofixa 3y agoThis is great news. The more projects do this, the less of a moat CUDA is, and the less of a competitive advantage Nvidia has.
- anonymous-panda 3y agoWhat does performance look like?
- sevagh 3y agoSpoiler alert: not good enough to break CUDA's moat
- qeternity 3y agoThis is not CUDA's moat. That is on the R&D/training side. Inference side is partly about performance, but mostly about cost per token. And given that there has been a ton of standardization around LLaMA architectures, AMD/ROCm can target this much more easily, and still take a nice chunk of the inference market for non-SOTA models.
- bornfreddy 3y agoNot sure why you're downvoted, but as far as I've heard AMD cards can't beat 4090 - yet. Still, I think AMD will catch or overtake NVidia in hardware soon, but software is a bigger problem. Hopefully the opensource strategy will pay off for them.
- arein3 3y agoReally hope so, maybe this time will catch and last Usually when corps open source stuff to get adoption, they stuff the adopters after they gain enough market share and the cycle repeats again
- nerdix 3y agoA RTX 4090 is about twice the price of and 50%-ish faster than AMD's most expensive consumer card so I'm not sure anyone really expects it to ever surpass a 4090. A 7900 XTX beating a RTX 4080 at inference is probably a more realistic goal though I'm not sure how they compare right now.
- Zambyte 3y agoThe 4080 is $1k for 16gb of VRAM, and the 7900 is $1k for 24gb of VRAM. Unless you're constantly hammering it with requests, the extra speed you may get with CUDA on a 4080 is basically irrelevant when you can run much better models at a reasonable speed.
- imtringued 3y agoHypotheticals don't matter. The average user won't have the most expensive GPU and when it comes to VRAM AMD is half as expensive so they lead in this area.
- Zambyte 3y agoI get 35tps on Mistral:7b-Instruct-Q6_K with my 6650 XT.
- ixaxaar 3y agoHey I did, and sorry for the self promo, Please check out https://github.com/geniusrise https://github.com/geniusrise - tool for running llms and other stuff, behaves like docker compose, works with whatever is supported by underlying engines: Huggingface - MPS, cuda VLLM - cuda, ROCm llama.cpp, whisper.cpp - cuda, mps, rocm Also coming up integration with spark (TorchDistributor), kafka and airflow.
- Symmetry 3y agoGiven the price of top line NVidia cards, if they can be had at all, there's got to be a lot of effort going on behind the scenes to improve AMD support in various places.
- deleted 3y ago[deleted]
- jeff-davis 3y agoWhat are the barriers to doing so?
- reilly3000 3y agoI’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.
- Apofis 3y agoI'm surprised they didn't speak about the implementation at all. Anyone got more intel?
- refulgentis 3y agoThey're open source and based on llama.cpp so nothings secret. My money, looking at nothing, would be on one of the two Vulkan backends added in Jan/Feb. I continue to be flummoxed by a mostly-programmer-forum treating ollama like a magical new commercial entity breaking new ground. It's a CLI wrapper around llama.cpp so you don't have to figure out how to compile it
- washadjeffmad 3y agoI tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI". And that more or less answered it.
- dartos 3y agoIt’s because most devs nowadays are new devs and probably aren’t very familiar with native compilation. So compiling the correct version of llama.cpp for their hardware is confusing. Compound that with everyone’s relative inexperience with configuring any given model and you have prime grounds for a simple tool to exist. That’s what ollama and their Modelfiles accomplish.
- tracerbulletx 3y ago
- deadalus 3y agoI wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450. AMD with ROCm is decent on Linux but pretty bad on Windows.
- choilive 3y agoThey bump up VRAM because they can't compete on raw compute.
- coconut08 3y agoit doesn't matter how much compute you have if you don't have enough vram to run the model.
- Zambyte 3y agoExactly. My friend was telling me that I was making a mistake for getting a 7900 XTX to run language models, when the fact of the matter is the cheapest NVIDIA card with 24 GB of VRAM is over 50% more expensive than the 7900 XTX. Running a high quality model at like 80 tps is way more important to me than running a way lower quality model at like 120 tps.
- mejutoco 3y agoI cannot parse this. The Radeon RX 7900 XTX also has 24GB of vram, so how does it help you run higher quality models? I would understand if it had more ram.
- Zambyte 3y agoOnly the RX 7900 XTX has 24 GB of VRAM at its price point. If I went with an NVIDIA card, I would either have to spend over 50% more on the card, or use much worse models to fit on their 16 GB cards.
- deleted 3y ago[deleted]
- simon83 3y agoDoes anyone know how the AMD consumer GPU support on Linux has been implemented? Must use something else than ROCm I assume? Because ROCm only supports the 7900 XTX on Linux[1], while on Windows[2] support is from RX 6600 and upwards. [1]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibility/linux-support.html#linux-supported-gpus https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi... [2]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibility/windows-support.html#windows-supported-gpus https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi...
- Symmetry 3y agoThe newest release, 6.0.2, supports a number of other cards[1] and in general people are able to get a lot more cards to work than are officially supported. My 7900 XT worked on 6.0.0 for instance. [1]https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...
- bravetraveler 3y agoI wouldn't read too much into support. It's more in terms of business/warranty/promises than what can actually do things I've had a 6900XT since launch and this is the first I'm hearing "unsupported", having played with ROCM plenty over the years with Fedora Linux. I think, at most, it's taken a couple key environment variables
- HarHarVeryFunny 3y agoHow hard would it be for AMD just to document the levels of support of different cards the way NVIDIA does with their "compute capability" numbers ?! I'm not sure what is worse from AMD - the ML software support they provide for their cards, or the utterly crap documentation. How about one page documenting AMD's software stack compared to NVIDIA, one page documenting what ML frameworks support AMD cards, and another documenting "compute capability" type numbers to define the capabilities of different cards.
- 3y ago
- KronisLV 3y agoFeels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.
- hugozap 3y agoI'm a happy user of Mistral on my Mac Air M1.
- isoprophlex 3y agoHow many gbs of RAM do you have in your M1 machine?
- hugozap 3y ago8gb
- isoprophlex 3y agoThanks, wow, amazing that you can already run a small model with so little ram. I need to buy a new laptop, guess more than 16 gb on a macbook isn't really needed
- dartos 3y agoMistral is _very_ small when quantized. I’d still go with 16gbs
- SparkyMcUnicorn 3y agoI would advise getting as much RAM as you possibly can. You can't upgrade later, so get as much as you can afford. Mine is 64GB, and my memory pressure goes into the red when running a quantized 70B model with a dozen Chrome tabs open.
- Zambyte 3y agoIt's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support. Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that they don't "officially" support yet. It's just a matter of setting an environment variable. [0] https://github.com/ollama/ollama/issues/2870 https://github.com/ollama/ollama/issues/2870 Edit: I did notice one change, now the starcoder2[1] model works now. Before that would crash[2]. [1] https://ollama.com/library/starcoder2 https://ollama.com/library/starcoder2 [2] https://github.com/ollama/ollama/issues/2953 https://github.com/ollama/ollama/issues/2953
- throwaway5959 3y agoI mean, it was 21 days ago. What’s the difference?
- yjftsjthsd-h 3y ago2 versions, apparently
- mchiang 3y agoWhile the PRs went in slightly earlier, much of the time was spent on testing the integrations, and working with AMD directly to resolve issues. There were issues that we resolved prior to cutting the release, and many reported by the community as well.
- Zambyte 3y agoThank you for clarifying and thanks for the great work you do!
- cjbprime 3y agoMaybe they wanted to wait for bug reports for 21 days before publishing a popular blog post about it..?
- SushiHippie 3y agoCan anyone (maybe ollama contributors) explain to me the relationship between llama.cpp and ollama? I always thought that ollama basically just was a wrapper (i.e. not much changes to inference code, and only built on top) around llama.cpp, but this makes it seem like it is more than that?
- Zambyte 3y agollama.cpp also supports AMD cards: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hipblas https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi...
- renewiltord 3y agoWow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.
- nottorp 3y agoUsed crypto mining parts not available any more?
- duskwuff 3y agoCrypto mining didn't require significant bandwidth to the card. Mining-oriented motherboards typically only provisioned a single lane of PCIe to each card, and often used anemic host CPUs (like Celeron embedded parts).
- renewiltord 3y agoExactly. They'd use PCIe x1 to PCIe x16 risers with power adapters. These require high-bandwidth.
- nottorp 3y agoOh. Shows I wasn't into that. I did once work with a crypto case, but yes, it was one motherboard with a lot of wifis and we still didn't need the pcie lanes.
- cjbprime 3y agoDoes LLM inference require significant bandwidth to the card? You have to get the model into VRAM, but that's a fixed startup cost, not a per-output-token cost.
- segmondy 3y agoYou don't need 128 lanes. 8x PCIe3 is more than enough, so for 4 cards that's 32. Most CPUs have about 40lanes. If you are not doing much that would be more than sufficient. Buy a PCIe riser. Go to amazon and search for it, a 16x to 16x PCIe riser. They go for about $25-$30 often about 20-30cm. If you want really long one, you can get one from China a 60cm for about the same price, you just have to wait for 3 weeks. That's what I did. Stuffing all those in a case is often difficult, so you have to go open rig. Either have the cables running out your computer and figuring out a way to support the cards while keeping them cool or just buy a small $20-$30 open rig frame.
- kodarna 3y agoI wonder why they aren't supporting RX 6750 XT and lower yet, are there architectural differences between these and RX 6800+?
- Zambyte 3y agoThey don't support it, but it works if you set an environment variable. https://github.com/ollama/ollama/issues/2870#issuecomment-1975477251 https://github.com/ollama/ollama/issues/2870#issuecomment-19...
- slavik81 3y agoThose are Navi 22/23/24 GPUs while the RX 6800+ GPUs are Navi 21. They have different ISAs... however, the ISAs are identical in all but name. LLVM has recently introduced a unified ISA for all RDNA 2 GPUs (gfx10.3-generic), so the need for the environment variable workaround mentioned in the other comment should eventually disappear.
- latchkey 3y agoIf anyone wants to run some benchmarks on MI300x, ping me.
- freedomben 3y agoI'm thrilled to see support for RX 6800/6800 XT / 6900 XT. I bought one of those for an outrageous amount during the post-covid shortage in hopes that I could use it for ML stuff, and thus far it hasn't been very successful, which is a shame because it's a beast of a card! Many thanks to ollama project and llama.cpp!
- mey 3y agoSad to see that the cut off is just after 6700 XT which is what is in my desktop. They indicate more devices are coming, hopefully that includes some of the more modern all in one chips with RDNA 2/3 from AMD as well.
- mey 3y agoIt appears that the cut off lines up with HIP SDK support from AMD, https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat...
- throawayonthe 3y agoI’ve already been using ollama with my 6700xt just fine, you just have to set some env variable to make rocm work “unoficially” The linked page says they will support more soon, so i’m guessing this will just be integrated
- Rising6378 3y agowhich env variable did you set to what? currently I'm struggeling setting it up
- Retr0id 3y agoHSA_OVERRIDE_GFX_VERSION=10.3.0 seems to be the magic words. You can invoke it directly via `HSA_OVERRIDE_GFX_VERSION=10.3.0 ollama serve`, but I added this line to the systemd unit at /etc/systemd/system/ollama.service: Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
- rcarmo 3y agoCurious to see if this will work on APUs. Have a 7840HS to test, will give it a go ASAP.
- airocker 3y agoI heard "Nvidia for LLM today is similar to how Sun Microsystems was for the web"
- rahimnathwani 3y agoHere is the commit that added ROCm support to llama.cpp back in August: https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560cb45dd2c51ad403bda8723b629 https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
- 65a 3y agoYep, and it deserves the credit! He who writes the cuda kernel (or translates it) controls the spice. I had wrapped this and had it working in Ollama months ago as well: https://github.com/ollama/ollama/pull/814 https://github.com/ollama/ollama/pull/814. I don't use Ollama anymore, but I really like the way they handle device memory allocation dynamically, I think they were the first to do this well.
- rahimnathwani 3y agoI'm curious about both: - what's special about the memory allocation, and how might it help me? - what are you now using instead of ollama?
- 65a 3y agoOllama does a nice job of looking at how much VRAM the card has and tuning the number of gpu layers offloaded. Before that, I mainly just had to guess. It's still a heuristic, but I thought that was neat. I'm mainly just using llama.cpp as a native library now, mainly for the direct access to more of llama's data structures, and because I have a sort of unique sampler setup.
- rahimnathwani 3y agoOh right... I've just been guessing, to try and find the value one fewer than the one which causes CUDA OOM errors.
- tarruda 3y agoDoes this work with integrated Radeon graphics? If so it might be worth getting one of those Ryzen mini PCs to use as a local LLM server.
- stebalien 3y agoNot yet: https://github.com/ollama/ollama/issues/2637 https://github.com/ollama/ollama/issues/2637
- eclectic29 3y agoI'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people are giving it. Edit: what do you do after the initial experimentation? you need to deploy these models eventually to production. I'm not even talking about giving credit to llama.cpp, just mentioning that this product is gaining disproportionate attention and kudos compared to the value it delivers. Not denying that it's a great product.
- reustle 3y agoEven for just running a model locally, Ollama provided a much simpler "one click install" earlier than most tools. That in itself is worth the support.
- Aka456 3y agoKoboldcpp is also very, very good, plug and play, very complet web UI, nice little api with sse text streaming, vulkan accelerated, have an AMD fork...
- crooked-v 3y ago> Granted that it makes the experimentation process super easy That's the answer to your question. It may have less space than a Zune, but the average person doesn't care about technically superior alternatives that are much harder to use.
- thejohnconway 3y ago*Nomad And lame.
- davidhariri 3y agoAs it turns out, making it faster and better to manage things tends to get people’s attention. I think it’s well deserved.
- observationist 3y agoThere's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty to have no shout out or thank you anywhere in the repository or blog or ollama website. GPT4All - https://gpt4all.io/index.html https://gpt4all.io/index.html LM Studio - https://lmstudio.ai/ https://lmstudio.ai/ Both of these projects credit and attribute appropriately, Ollama seems to bend over backwards so they don't have to?
- jart 3y agoollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had great success focusing on things that wouldn't make sense for llama.cpp. Besides everyone who wants to know what's up can read the source code, research papers, etc. then make their own judgements about who's who. It's all in the open.
- refulgentis 3y agoLike what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt. And they could have very, very, easily, there's a server, sitting right there. They chose not to. That's fine. But it's a choice. The rest I chalk it up to inexperience and being busy.
- jjice 3y agoJust downloaded this and gave it a go. I have no experience with running any local models, but this just worked out of the box on my 7600 on Ubuntu 22. This is fantastic.
- haolez 3y agoIs there an equivalent to ollama or gpt4all for Android? I'd like to host my model somewhere and talk to it via app.
- verdverm 3y agoAt that point, it sounds like your after an API endpoint for any model. Lots of solutions out there, depends more on your hosting
- haolez 3y agoNo, I want the opposite: I want a generic chatbot app that connects to any endpoint.
- verdverm 3y agoAh, that's not what ollama is, it runs models for you, it doesn't talk to them over an API I would imagine someone has built an app that can talk to LLM apis
- BOOSTERHIDROGEN 3y agoOllama can serve as an endpoint.
- treprinum 3y agoAny info on the performance? How does it compare to 4080/4090?
- OtomotO 3y agoHm, fooocus manages to run, but for Ollama I get: >time=2024-03-16T00:11:07.993+01:00 level=WARN source=amd_linux.go:50 msg="ollama >recommends running the https://www.amd.com/en/support/linux-drivers https://www.amd.com/en/support/linux-drivers: amdgpu version file >missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or >directory" >time=2024-03-16T00:11:07.993+01:00 level=INFO source=amd_linux.go:85 msg="detected amdgpu >versions [gfx1031]" >time=2024-03-16T00:11:07.996+01:00 level=WARN source=amd_linux.go:339 msg="amdgpu >detected, but no compatible rocm library found. Either install rocm v6, or follow manual >install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual->install https://github.com/ollama/ollama/blob/main/docs/linux.md#man..." >time=2024-03-16T00:11:07.996+01:00 level=WARN source=amd_linux.go:96 msg="unable to verify >rocm library, will use cpu: no suitable rocm found, falling back to CPU" >time=2024-03-16T00:11:07.996+01:00 level=INFO source=routes.go:1105 msg="no GPU detected" Need to check how to install rocm on arch again... have done it once, a few moons back, but alas...
- jmorgan 3y agoAh, this is probably from missing ROCm libraries. The dynamic libraries are available as one of the release assets (warning: it's about 4GB expanded) https://github.com/ollama/ollama/releases/tag/v0.1.29 https://github.com/ollama/ollama/releases/tag/v0.1.29 – dropping them in the same directory as the `ollama` binary should work.
- deleted 3y ago[deleted]
- h4x0rr 3y agoNo support for my rx 5700xt :(
- userbinator 3y agoThanks, Ollama. (Sorry, could not resist.)
- ekianjo 3y agoDoes it also support AMD APUs?
- VadimPR 3y agoOllama runs really, really slow on my MBP for Mistral - as in just a few tokens a second and it takes a long while before it starts giving a result. Anyone else run into this? I've seen that it seems to be related to the amount of system memory available when ollama is started (??) however LM Studio does not have such issues.
- skenderbeu 3y agollama.cpp now supports AMD graphics. Ollama just wraps around it
- avocardio 3y ago@geohot