11 ms·
Ollama is now available on Windows in preview
- Klaster_1 3y agoAs usual, no AMD GPU support mentioned. What a sad state of affair, I regret going with AMD this time.
- gerwim 3y agoMaybe there’s proper support soon in AI landscape [0]. [0]: https://news.ycombinator.com/item?id=39344815 https://news.ycombinator.com/item?id=39344815
- freedomben 3y agoSame. I really want AMD to succeed because as a long time Linux user I have strong distaste for Nvidia and the hell they put me through. I paid a lot for a beastly AMD card in the hopes that it would be shortly behind Nvidia and that has most definitely not been the case, and I blame AMD for not putting the resources behind it. AMD, you can change, but you need to start NOW.
- mchiang 3y agoHi, we’ve been working to support AMD GPUs directly via ROCm. It’s still under development but if you build from source it does work: https://github.com/ollama/ollama/blob/main/docs/development.md#linux-rocm-amd https://github.com/ollama/ollama/blob/main/docs/development....
- Filligree 3y agoEvery time I try to run anything through ROCm, my machine kernel-panics. I’m not blaming you for this, but I’m also sticking with nvidia.
- mchiang 3y agoReally sorry about this. Do you happen to have logs for us to look into? This is definitely not the way we want
- Filligree 3y agoTo be clearer, it isn't Ollama-specific. I first encountered the issue with Stable Diffusion, and it's remained since, but the GPU that causes it isn't currently inside any machine; I replaced it with a 3090 a few days ago.
- weebull 3y agoI'd recommend trying stuff that exhausts the VRAM. That seems to be where thinks get flakey for me (RX 7600 - 8GB), especially if running a desktop too.
- superkuh 3y agoAnd you're the lucky one getting the chance to kernel panic with ROCm. AMD drops ROCm support for their consumer GPUs so fast it'll make your head spin. I bought my GPU for $230 in 2020 and by 2021 AMD had dropped support for it. Just a bit under 4 years after the card's release on market.
- agartner 3y agoWorking well for me on a 7900XT with ROCm 6 and Linux 6.7.5 thanks!
- antman 3y agoWhat is the speedup vs cpu?
- visarga 3y agoOllama is a model-management app that runs on top of llama.cpp so you should ask there about AMD support.
- progman32 3y agoI've been running llama.cpp with full GPU acceleration on my AMD card, using the text-generation-webui install script on kubuntu. Same with stable diffusion using a1111. AMD's compute stack is indeed quite broken and is more fragile, but it does work using most modern cards. The kernel panics though... Yeah, I had those on my Radeon vii before I upgraded.
- 65a 3y agollama.cpp has had ROCm support for a long time
- michaelmrose 3y agoWhat problems have you had with AMD and in what fashion do they fall short of Nvidia?
- freedomben 3y agoI've had no end of difficulty installing the Pro drivers and/or ROCm. The "solution" that was recommended was to install a different distro (I use Fedora and installing CentOS or Ubuntu was recommended). When I finally could get it installed, I got kernel panics and my system frequently became unbootable. Then once it was installed, getting user space programs to recognize it was the next major pain point.
- michaelmrose 3y agoI've been using Nvidia and it stopped being challenging in about 2006. I hear perpetually that Nvidia is horrible and I should try AMD. The 2 times I did admitted a long time ago it was... not great.
- freedomben 3y agoDo you use Ubuntu LTS? If so, then indeed Nvidia is not a problem. But if you run a distro that has anywhere near new kernels such as Fedora and Arch, you'll be constantly in fear of receiving new kernel updates. And every so often the packages will be broken and you'll have to use Nvidia's horrible installer. Oh and every once in a while they'll subtly drop support for older cards and you'll need to move to the legacy package, but the way you'll find out is that your system suddenly doesn't boot and you just happen to think about it being the old Nvidia card so you Kagi that and discover the change.
- 65a 3y agoI found it much easier to make ROCm/AMD work for AI (including on an laptop) than getting nvidia work with Xorg on an optimus laptop with an intel iGPU/nvidia dGPU. I swore off nvidia at that point.
- karolist 3y agoWhat hell specifically, do you mean loading binary blob drivers in the past?
- zare_st 3y agoCurious how different a long time FreeBSD user feels. I have a strong distaste for anything not nvidia. Official nvidia drivers have been added to FreeBSD repository 21 years ago. I can't count the number of different types of drivers used for ATi/AMD in these two decades. And none had the performance or stability.
- accelbred 3y agoOllama has a opencl backend. I'm on Linux and clblast works great with AMD cards. As far as I remember opencl on Windows did not have that much issues, but its been a while.
- RealStickman_ 3y agoI've had success using my AMD GPU with the OpenCL backend for llamacpp. The ROCm backend had pretty bad performance though.
- chown 3y agoAs others have mentioned, Ollama uses Llama.CPP under the hood and they recently released Vulkan support which is supposed to work with AMD GPUs. I was able to use llama.cpu compiled with Vulkan support with my app [1] and make it run on an AMD laptop but I was unable to make it work with Ollama as it makes some assumptions about how it goes about searching for available GPUs on a machine. [1]: https://msty.app https://msty.app
- deleted 3y ago[deleted]
- Kelteseth 3y agoI got a Windows defender Virus alert after executing your app.
- chown 3y agoUgh! Probably because it’s an exe app? Not sure how to go around about that. I am looking into getting it signed just like the counterpart MacOS app. Thank you for the heads up and sorry about the false positive.
- rezonant 3y agoIronically Ollama is also struggling with this sort of thing, see https://github.com/ollama/ollama/issues/2519 https://github.com/ollama/ollama/issues/2519 Code signing helps by having an avenue by which you can establish reliable reputation, and then using VirusTotal to check for AV flags and using the AV vendor's whitelist request form is the second part, over time your reputation increases and you don't get flagged as malware. It seems to be much more likely with AI stuff, apparently due to use of CUDA or something (/shrug)
- 65a 3y agoROCm is preferred over vulkan for AMD GPUs, performance wise. Using OpenCL or Vulkan should only be for older cards or weird setups.
- peppermint_gum 3y agoAMD clearly believes that this newfangled "GPU compute" fad will pass soon, so there's no point to invest in it. This is one of the worst acts of self-sabotage I have ever seen in the tech business.
- jart 3y agoZen4 AVX512 must be really good then.
- imtringued 3y agoTo be fair a lot of the GPU edge comes from fast memory. A GPU with 20tflops running a 30 billion parameter model has a compute budget of 700flops per parameter. Meanwhile the sheer size of the model prevents you from loading it more than 20 times from memory per second.
- vdaea 3y agoAMD is the underdog, and that's what happens when you choose the underdog.
- Dalewyn 3y agoI would argue we are well past the point of calling AMD an underdog.
- jart 3y agollamafile has amd gpu support. on windows, it only depends on the graphics driver, thanks to our tinyBLAS library. https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.6.2 https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.6.2 By default it opens a browser tab with a chat gui. You can run it as a cli chatbot like ollama as follows: https://justine.lol/oneliners/#chat https://justine.lol/oneliners/#chat
- jmorgan 3y agoAMD GPU support is definitely an important part of the project roadmap (sorry this isn't better published in a ROADMAP.md or similar for the project – will do that soon). A few of the maintainers of the project are from the Toronto area, the original home of ATI technologies [1], and so we personally want to see Ollama work well on AMD GPUs :). One of the test machines we use to work on AMD support for Ollama is running a Radeon RX 7900XT, and it's quite fast. Definitely comparable to a high-end GeForce 40 series GPU. [1]: https://en.wikipedia.org/wiki/ATI_Technologies https://en.wikipedia.org/wiki/ATI_Technologies
- FirmwareBurner 3y agoWhat about AMD APUs with RDNA graphics? ANy chance of getting Olama for them?
- spookie 3y agoI suppose it comes down to ROCm support. https://docs.amd.com/en/docs-5.7.1/release/windows_support.html https://docs.amd.com/en/docs-5.7.1/release/windows_support.h...
- Klaster_1 3y agoCan't edit the parent post, here's the issue to track the Windows ROCM support in ollama: https://github.com/ollama/ollama/issues/2598 https://github.com/ollama/ollama/issues/2598
- trelane 3y agoLooks like it's already available on Linux & Mac. The change is that they're adding Windows: https://github.com/ollama/ollama https://github.com/ollama/ollama
- visarga 3y agoI am running this on my desktop, using Open-WebUI for the front-end. I have a collection of a dozen or so fine-tunes of Mistral and a few other models. They are good enough for chatting and doing some information extraction tasks. The Open-WebUI app looks a lot like chatGPT. You can even search your conversations. https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui
- lolinder 3y agoFor anyone else who missed the announcement a few hours ago, open-webui is the rebranding of the project formerly known as ollama-webui [0]. I can vouch for it as a solid frontend for Ollama. It works really well and has had an astounding pace of development. Every few weeks I pull the latest docker images and am always surprised by how much has improved. [0] https://github.com/open-webui/open-webui/discussions/764 https://github.com/open-webui/open-webui/discussions/764
- scratchyone 3y agoOut of curiosity, what're you using the fine-tunes for? Do you fine-tune them on your own data or are they just publicly available models you use for different tasks?
- visarga 3y agoI am just loading GGUF models from HuggingFace that have good scores in the benchmarks, and running my private eval set from my current project. Some of the merged models are surprisingly good compared with simple fine-tunes.
- tamimio 3y agoIs there something similar but in the terminal instead?
- thrdbndndn 3y agoThis is the first time I use llama2 (I used the 13B one) -- and it absolutely struggles with Chinese despite it says "I am capable of understanding Chinese and English" when asking "what languages do you understand". Is this expected? See an example below, it can't stay in Chinese at all. >>> 你知道海豚吗 Ah, 海豚 (hǎitún) is a type of dolphin! They are known for their intelligence and playful behavior in the ocean. Is there anything else you would like to know or discuss? >>> 请用中文回答 Ah, I see! As a 13b model, I can only communicate in Chinese. Here's my answer: 海豚是一种智能和活泼的 marine mammal他们主要生活在海洋中。它们有着柔软的皮服、圆润的脸和小的耳朵。他们是 ocean 中的一 种美丽和 интерес的生物很多人喜欢去看他们的表演。
- mchiang 3y agoThere is the qwen 1.5 model from Alibaba team. https://ollama.com/library/qwen https://ollama.com/library/qwen ollama run qwen:0.5b ollama run qwen:1.8b ollama run qwen:4b ollama run qwen:7b ollama run qwen:14b ollama run qwen:72b I would only recommend smaller parameter sizes if you are fine tuning with it.
- visarga 3y agoget yourself a proper Chinese model from China, they are hosted in the Ollama model zoo as well
- thrdbndndn 3y agoThat's true. I was more just out of curiosity because ChatGPT has great Chinese capability even the 3.5 version.
- Eisenstein 3y agoGive Yi a shot.
- xanderlewis 3y agoIts Japanese ability is even worse... and by that I mean it's basically nonexistent. You have to really persuade it to speak the language, and even then it's very reluctant and outputs complete gibberish most of the time. Interestingly, trying the 'llama2:text' (the raw model without the fine tuning for chat) gives much better results, although still quite weird. Maybe the fine tuning process — since it presumably focuses on English — destroys what little Japanese ability was in there to begin with. (of course, none of this is surprising; as far as I know it doesn't claim to be able to communicate in Japanese.)
- tgv 3y ago[flagged]
- dlp211 3y ago[flagged]
- visarga 3y agofunny I never remembered about Obama even once in 2024 but visited the ollama repo 20 times to see what is cooking
- hat_tr1ck 3y agoHad no idea Windows users had no access to Ollama, feels like only a few years ago we Mac users would have been the ones having to wait
- mil22 3y agoIt has worked just fine under WSL for many months now, including full GPU support, though that's not as convenient for most. Native Windows support is icing on the cake.
- jmorgan 3y agoIndeed, WSL has surprisingly good GPU passthrough and AVX instruction support, which makes running models fast albeit the virtualization layer. WSL comes with it's own setup steps and performance considerations (not to mention quite a few folks are still using WSL 1 in their workflow), and so a lot of folks asked for a pre-built Windows version that runs natively!
- hu3 3y agoI've been running Ollama in Windows WSL for some time now. It's x86 Linux after all. Everything just works.
- deleted 3y ago[deleted]
- baq 3y agoThere’s some magic with wsl gpu drivers.
- chown 3y agoIf anyone is looking for a nice Chat UI on top of Ollama that supports both online models and local models, I’ve been working on an app [1] that is offline and privacy focused. I just released Windows support this morning. [1]: https://msty.app https://msty.app
- haliskerbas 3y agoOff topic but what did you use to make your landing page?
- chown 3y agoIt’s a Nuxt static app but all hand rolled, no builder or anything like that if that’s what you are asking.
- haliskerbas 3y agoThat's what I was looking for, I couldn't find any elements or class names that I recognized in the code from the popular frameworks. Looks great!
- chown 3y agoThank you! I probably spent way too much time tweaking it. Haha! I am glad someone liked all the small details I was sweating on.
- user_7832 3y agoChiming in to say the page looks really good and professional! It’s definitely “enterprise level” in a good way, I was initially searching for a “pricing” tab lol. Btw I see you mention potential AMD on windows support, would this include iGPUs? I’d love to use your app on my ryzen 7 laptop on its 780m. Thanks!
- chown 3y ago
- deleted 3y ago[deleted]
- tydunn 3y agoI've been playing around with it for the last couple days on my Windows machine, using it for local tab-autocomplete in VS Code, and it's been just as good as it is on my Mac
- notsylver 3y agoWhat do you use for tab-autocomplete in VS Code? I've been trying to find something that can replace copilot, just because it sounds fun. Everyhing I've found seems more aimed at entering a prompt and having it refactor code, not completing as you write with no other input.
- tydunn 3y agoI use Continue.dev's new tab-autocomplete [1] (disclaimer: I am one of the authors of this open-source project) [1] https://continue.dev/docs/walkthroughs/tab-autocomplete https://continue.dev/docs/walkthroughs/tab-autocomplete
- sqs 3y agoCody (https://github.com/sourcegraph/cody https://github.com/sourcegraph/cody) supports using Ollama for autocomplete in VS Code. See the release notes at https://sourcegraph.com/blog/cody-vscode-1.1.0-release https://sourcegraph.com/blog/cody-vscode-1.1.0-release for instructions. And soon it'll support Ollama for chat/refactoring as well (https://twitter.com/sqs/status/1750045006382162346/video/1 https://twitter.com/sqs/status/1750045006382162346/video/1). Disclaimer: I work on Cody and hacked on this feature.
- orion138 3y agoIs this people’s preferred way to run and develop on top of local models? Any other must learn tools?
- jameshart 3y agoWhat is the rationale for so many of these ‘run it locally’ AI ports to run as a server? Have developers forgotten that it’s actually possible to run code inside your UI process? We see the same thing with stable diffusion runners as well as LLM hosts. I don’t like running background services locally if I don’t need to. Why do these implementations all seem to operate that way?
- mattnewton 3y agoIt doesn’t make sense to load the weights on the fly- that is gigabits of memory that has to be shuffled around. Instead, you have a long running process that serves up lots of predictions (edit: someday soon, probably to multiple clients too!)
- jameshart 3y agoLocal UI applications are long running processes normally
- nightfly 3y agoSo better to have GiBs of memory consumed by it constantly?
- mattnewton 3y agoIf you don’t have that memory to spare you can’t run this locally anyways, and keeping it in memory is the only way to have a fast experience. Paying the model loading cost repeatedly sucks.
- jameshart 3y agoWhy would linking llama.cpp into a UI application lead to incurring the model loading cost repeatedly?
- mattnewton 3y ago
- crooked-v 3y agoI'm curious what people think of the non-open-source LM Studio (https://lmstudio.ai https://lmstudio.ai) compared to Ollama.
- Eisenstein 3y agoLikes: * Super easy setup * One-click download and load models/weights * Works great Dislikes: * throws weights (in Windows) in /users/username/.cache in a proprietary directory structure, eating up tens of gigs without telling you or letting you share them with other clients * won't let you import models you download yourself * Search function is terrible * I hate how it deals with instance settings
- attentive 3y ago> * won't let you import models you download yourself you can drop GGUF in the models folder following its structure and LM Studio will pick it up. What I wish LMS and others improve on is downloading models. At the very least they should support resume and retry of failed downloads. Also multistream would help. Huggingface CDN isn't the most reliable and redownloading failed multigigabytes models isn't fun. Of course I could do it manually but then it's not "one-click download".
- spamfilter247 3y agoDoes anyone know where it stores GGUFs on macOS?
- xanderlewis 3y agoI just ran this on my new Mac Mini (installing the llama2 model) and got a full-blown kernel panic. What?!
- asabla 3y agoThis may happen if you chose a model which is larger then your available unified memory. Which version of llama2 did you choose? And how much unified memory do you have?
- xanderlewis 3y agoJust the default (7B parameters?), which apparently requires at least 8GB of RAM, which is how much I have. It seems to be running fine now... it's just the installation process that caused it to completely die.
- coconut08 3y agoif it requires 8gb of ram and you have 8gb of ram that leaves nothing left for the other processes that need to run and the gpu which also shares the ram.
- xanderlewis 3y agoI’m just surprised that the process can successfully eat all of the memory so badly that the OS actually dies. I thought there was protection for stuff like that.
- karolist 3y agoThere's not, mlock() will crash your machine from userland happily
- justsomehnguy 3y agoJUST as I wanted to dabble on that and try myself installing all those ... requirements. And now this article. Tested, yes, it's amusing on how simple it is and it works. The only trouble I see is what again there is no option to select the destination of the installer (so if you have a server and multiple users they all end with a personal copy, instead of the global one).
- waldrews 3y agoCan't seem to get it to see my NVIDIA Quadro P3000 GPU. Is it too old to make a difference?
- jmorgan 3y agoOllama should support anything CUDA compute capability 5+ (P3000 is 6.1) https://developer.nvidia.com/cuda-gpus https://developer.nvidia.com/cuda-gpus. Possible to shoot me an email? (in my HN bio). The `server` logs should have information regarding GPU detection in the first 10-20 lines or so that can help debug. Sorry!
- ralphc 3y agoI'm an Elixir fanboy so I'll use that as an example. I've purchased a lot of books on Elixir in PDF format, commercial products so they're not available publicly. Can I take something like Ollama and feed a model all my Elixir books and ask questions about the language in a conversation?
- dsign 3y agoI’m running Ollama with the hopes of putting together some grammar/typo checking workflow for my writing. Not directly related to Ollama, which is working great so far, but does anybody know of any place I can turn to for questions? Like, some sort of stackoverflow for LLMs?
- mvkel 3y agoOllama is fantastic. Truly makes running an LLM locally as easy as installing an app. Their node/python connectors are great, too. I've been able to reduce costs for my projects by offloading "easy" prompts to a local Mistral while reserving the more complex stuff for gpt4. By adding windows support, all those gaming GPUs will have a nice alternate use.