5 ms·
Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
- nijave 6mo agoAnyone compare to ollama? I had good success with latest ollama with ROCm 7.4 on 9070 XT a few days ago
- iugtmkbdfil834 6mo agoSeconded. Currently on ollama for local inference, but I am curious how it compares.
- LumielGR 6mo agoLemonade is using llama.cpp for text and vision with a nightly ROCm build. It can also load and serve multiple LLMs at the same time. It can also create images, or use whisper.cpp, or use TTS models, or use NPU (e.g Strix Halo amdxdna2), and more!
- RealFloridaMan 6mo agoIt is optimized for compatibility across different APIs as well as has specific hardware builds for AMD GPUs and NPUs. It’s run by AMD. Under the hood they are both running llama.cpp, but this has specific builds for different GPUs. Not sure if the 9070 is one, I am running it on a 370 and 395 APU.
- metalliqaz 6mo agobetter than Vulkan?
- cpburns2009 6mo agoIn my experience using llama.cpp (which ollama uses internally) on a Strix Halo, whether ROCm or Vulkan performs better really depends on the model and it's usually within 10%. I have access to an RX 7900 XT I should compare to though.
- metalliqaz 6mo agoPerhaps I should just google it, but I'm under the impression that ollama uses llama.cpp internally, not the other way around. Thanks for that data point I should experiment with ROCm
- cpburns2009 6mo agoI meant ollama uses llama.cpp internally. Sorry for the confusion.
- naasking 6mo agoFrom what I understand, ROCm is a lot buggier and has some performance regressions on a lot of GPUs in the 7.x series. Vulkan performance for LLMs is apparently not far behind ROCm and is far more stable and predictable at this time.
- hrmtst93837 6mo ago[flagged]
- metalliqaz 6mo agoI was talking about ROCm vs Vulkan. On AMD GPUs, Vulkan has been commonly recognized as the faster API for some time. Both have been slower than CUDA due to most of the hosting projects focusing entirely on Nvidia. Parent post seemed to indicate that newer ROCm releases are better.
- naasking 6mo agoYes, Vulkan is currently faster due to some ROCm regressions: https://github.com/ROCm/ROCm/issues/5805#issuecomment-4141615579 https://github.com/ROCm/ROCm/issues/5805#issuecomment-414161... ROCm should be faster in the end, if they ever fix those issues.
- 0x457 6mo agoFor me Vulkan performs better on integrated cards, but ROCm (MIGraphX) on 7900 XTX.
- nijave 6mo agoAs I understand it, it depends on your GPU and ROCm version but they're similar-ish
- martin-adams 6mo agoI just compared this on my Mac book M1 Max 64GB RAM with the following: Model: qwen3.59b Prompt: "Hey, tell me a story about going to space" Ollama completed in about 1:44 Lemonade completed in about 1:14 So it seems faster in this very limited test.
- nezhar 6mo agoI'm also curious about this one, also I want to compare this to vLLM.
- 9dc 6mo agoso... what does it do? i dont get it Lol
- iugtmkbdfil834 6mo agoInitial read suggests it is a mini-swiss army knife, because it seems to be able to do a lot ( based on website claims anyway ). The app integration seems to suggest they want to be more of a control dashboard.
- syntaxing 6mo agoWow this is super interesting. This creates a local “Gemini” front end and all. This is more or less a generative AI aggregator where it installs multiple services for different gen modes. I’m excited to try this out on my strix halo. The biggest issue I had is image and audio gen so this seems like a great option.
- jmillikin 6mo agoSurprising that the Linux setup instructions for the server component don't include Docker/Podman as an option, its Snap/PPA for Ubuntu and RPM for Fedora. Maybe the assumption is that container-oriented users can build their own if given native packages?
- freedomben 6mo agoThey do have some container options, though I definitely think they should be added to the release page: https://lemonade-server.ai/install_options.html#docker https://lemonade-server.ai/install_options.html#docker
- zenoprax 6mo agoWhy should this be on the "Releases"? Shouldn't that just be for build artifacts? Pre-built containers belong on a registry, no? I suppose a Dockerfile could be included but that also seems unconventional.
- freedomben 6mo agoI just meant on the instructions part of the releases page (since they already have some installation instructions), not the artifacts themselves.
- freedomben 6mo agoNeat, they have rpm, deb, and a companion AppImage desktop app[1]! Surprised I wasn't aware of this project before. Definitely going to give it a try. [1]: https://github.com/lemonade-sdk/lemonade/releases/tag/v10.0.1 https://github.com/lemonade-sdk/lemonade/releases/tag/v10.0....
- JSR_FDED 6mo agoI’ve read the website and the news announcement, and I still don’t understand what it is. An alternative to LM Studio? Does it support MLX or metal on Macs? I’m assuming it will optimize things for AMD, but are you at a disadvantage using other GPUs?
- zelphirkalt 6mo agoI think LM Studio itself uses other software to actually make use of LLMs. If that other software does not support your NPUs, then you are not going to get much performance out of those. This Lemonade thing I am guessing is one such other software, that LM Studio could be using.
- molticrystal 6mo ago>Does it support MLX or metal on Macs? This is answered from their Project Roadmap over on Github[0]: Recently Completed: macOS (beta) Under Development: MLX support [0] https://github.com/lemonade-sdk/lemonade?tab=readme-ov-file#project-roadmap https://github.com/lemonade-sdk/lemonade?tab=readme-ov-file#...
- RealFloridaMan 6mo agoIt’s an easy way to get started and maintain a local AI stack that concentrates on AMD optimization. It is a one stop install for endpoints for sst, tts, image generation, and normal LLM. It has its own webui for management and interacting with the endpoints. It also has endpoints that are compatible with OpenAI, Ollama, and Anthropic so you can throw any tool that is compatible with those and it will just run.
- 0x457 6mo agoIt's alternative to LM Studio in a way that it's an abstraction over multiple runtimes. AMD part is that it supports FastFlowML runtime which is the only way to utilize NPU on Ryzen AI CPUs on linux.
- zozbot234 6mo agoNote that the NPU models/kernels this uses are proprietary and not available as open source. It would be nice to develop more open support for this hardware.
- swiftcoder 6mo agoAre they? The docs say "You can also register any Hugging Face model into your Lemonade Server with the advanced pull command options"
- zozbot234 6mo agoThat won't give you NPU support, which relies on https://github.com/FastFlowLM/FastFlowLM https://github.com/FastFlowLM/FastFlowLM . And that says "NPU-accelerated kernels are proprietary binaries", not open source.
- plagiarist 6mo agoI bought one of their machines to play around with under the expectation that I may never be able to use the NPU for models. But I am still angry to read this anyway.
- zozbot234 6mo agoAMD/Xilinx's software support for the NPU is fully open, it's only FFLM's models that are proprietary. See https://github.com/amd/iron https://github.com/amd/iron https://github.com/Xilinx/mlir-aie https://github.com/Xilinx/mlir-aie https://github.com/amd/RyzenAI-SW/ https://github.com/amd/RyzenAI-SW/ . It would be nice to explore whether one can simply develop kernels for these NPU's using Vulkan Compute and drive them that way; that would provide the closest unification with the existing cross-platform support for GPU's.
- moconnor 6mo agoIs... is this named because they have a lemon they're trying to make the most of?
- deleted 6mo ago[deleted]
- TeMPOraL 6mo agoIf life keeps giving it them, they should instead invent a combustible lemon.
- eddieroger 6mo agoDo they know who you are? They're the guys who are going to blow your house up ... with the lemons.
- LorenDB 6mo agoOn an unrelated note, do you think this software supports running models from a CD?...
- deleted 6mo ago[deleted]
- parsimo2010 6mo agoI think saying "L-L-M" sounds kind of like "lemon," so this is an LLM-aid (sounds like lemonade).
- metalliqaz 6mo agoso obvious and yet I didn't connect the dots. thank you
- ProllyInfamous 6mo agowait until you discover the LuLuleMonade -connection /s
- sensitiveCal 6mo agoFeels like this is sitting somewhere between Ollama and something like LM Studio, but with a stronger focus on being a unified “runtime” rather than just model serving. The interesting part to me isn’t just local inference, but how much orchestration it’s trying to handle (text, image, audio, etc). That’s usually where things get messy when running models locally. Curious how much of this is actually abstraction vs just bundling multiple tools together. Also wondering if the AMD/NPU optimizations end up making it less portable compared to something like Ollama in practice.
- RealFloridaMan 6mo agoIt bundles tools, model selection, and overall management. It’s portable in the sense it will install on any of the supported OS using CPU or vulkan backends. But it only supports out of the box ROCM builds and AMD NPUs. There is a way to override which llama.cpp version it uses if you want to run it on CUDA, but that adds more overhead to manage. If you have an AMD machine and want to run local models with minimal headache…it’s really the easiest method. This runs on my NAS, handles my home assistant setup. I have a strix halo and another server running various CUDA cards I manage manually by updating to bleeding edge versions of llama.cpp or vllm.
- ilaksh 6mo agoCool but is there a reason they can't just make PRs for vLLM and llama.cpp? Or have their own forks if they take too long to merge?
- RealFloridaMan 6mo agoThey use the latest llama.cpp under the hood but built for specific AMD GPU hardware. Lemonade is really just a management plane/proxy. It translates ollama/anthropic APIs to OpenAI format for llama.cpp. It runs different backends for sst/tts and image generation. Lets you manage it all in one place.
- cpburns2009 6mo agoJust in case anyone isn't aware. NPUs are low power, slow, and meant for small models.
- jcgrillo 6mo agoI wonder what was the imagined use case? TBH I was seriously thinking about buying a framework desktop but the NPU put me off.. I don't get why I should have to pay money for a bunch of silicon that doesn't do anything. And now that there's some software support... it still doesn't do anything? Why does it even exist at all then?
- naasking 6mo agoSmall models aren't entirely useless, and the NPU can run LLMs up to around 8B parameters from what I've seen. So one way they could be useful: Qwen3 text to speech models are all under 2B parameters, and Open AI's whisper-small speech to text model is under 1B parameters, so you could have an AI agent that you could talk to and could talk back, where, in theory, you could offload all audio-text and text-audio processing to the low power NPU and leave the GPU to do all of the LLM processing.
- zozbot234 6mo agoYou could always offload some layers to the NPU for lower power use and leave the rest to the GPU. If the latter is power throttled (common for prefill, not for decode) that will be a performance improvement.
- naasking 6mo agoRouting in a MoE model might fit.
- zozbot234 6mo agoYou want routing to be as quick as possible, because there are dependent loads of expert MoE weights (at least from CPU in most setups, potentially from storage) downstream of it. So that ultimately depends on what the bottleneck on that part of the model is: compute, memory throughput or both? If it's throughput, the NPU might be a bad fit.
- kouunji 6mo agoI’m looking forward to trying this currently Strix halo’s npu isn’t accessible if you’re running Linux, and previously I don’t think lemonade was either. If this opens up the npu that would be great! Resolute raccoon is adding npu support as well.
- dennemark 6mo agoMaybe you have seen NPU support via FLM already: https://lemonade-server.ai/flm_npu_linux.html https://lemonade-server.ai/flm_npu_linux.html "FastFlowLM (FLM) support in Lemonade is in Early Access. FLM is free for non-commercial use, however note that commercial licensing terms apply. "
- boomskats 6mo agoI thought the NPU has been available since something like 6.12?
- cpburns2009 6mo agoThe NPU works on Linux (Arch at least) on Strix Halo using FastFlowLM [1]. Their NPU kernels are proprietary though (free up to a reasonable amount of commercial revenue). It's neat you can run some models basically for free (using NPU instead of CPU/GPU), but the performance is underwhelming. The target for NPUs is really low power devices, and not useful if you have an APU/GPU like Strix Halo. [1]: https://github.com/FastFlowLM/FastFlowLM https://github.com/FastFlowLM/FastFlowLM
- dennemark 6mo agoI have been using lemonade for nearly a year already. On Strix Halo I am using nothing else - although kyuz0's toolboxes are also nice (https://kyuz0.github.io/amd-strix-halo-toolboxes/ https://kyuz0.github.io/amd-strix-halo-toolboxes/) Nowadays you get TTS, STT, text & image generation and image editing should also be possible. Besides being able to run via rocm, vulkan or on CPU, GPU and NPU. Quite a lot of options. They have a quite good and pragmatic pace in development. Really recommend this for AMD hardware! Edit: OpenAI and i think nowaday ollama compatible endpoints allow me to use it in VSCode Copilot as well as i.e. Open Web UI. More options are shown in their docs.
- syntaxing 6mo agoHave you used it with any agents or claw? If so, which model do you run?
- dennemark 6mo agoI have two Strix Halo devices at hand. Privately a framework desktop with 128gb and at work 64GB HP notebook. The 64GB machine can load Qwen3.5 30B-A3B, with VSCode it needs a bit of initial prompt processing to initialize all those tools I guess. But the model is fighting with the other resources that I need. So I am not really using it anymore these days, but I want to experiment on my home machine with it. I just dont work on it much right now. Lemonade has a Web UI to set the context size and llama.cpp args, you need to set context to proper number or just to 0 so that it uses the default. If its too low, it wont work with agentic coding. I will try some Claw app, but first need to research the field a bit. But I am using different models on Open Web UI. GPT 120B is fast, but also Qwen3.5 27B is fine.
- cpburns2009 6mo agoQwen3-Coder-Next works well on my 128GB Framework Desktop. It seems better at coding Python than Qwen3.5 35B-A3B, and it's not too much slower (43 tg/s compared to 55 tg/s at Q4). 27B is supposed to be really good but it's so slow I gave up on it (11-12 tg/s at Q4).
- rpdillon 6mo agoBeen running lemonade for some time on my Strix Halo box. It dispatches out to other backends that they include, like diffusion and llama. I actually don't like their combined server, and what I use instead is their llama CPP build for ROCm. https://github.com/lemonade-sdk/llamacpp-rocm https://github.com/lemonade-sdk/llamacpp-rocm But I'm not doing anything with images or audio. I get about 50 tokens a second with GPT OSS 120B. As others have pointed out, the NPU is used for low-powered, small models that are "always on", so it's not a huge win for the standard chatbot use case.
- zozbot234 6mo agoEven small NPUs can offload some compute from prefill which can be quite expensive with longer contexts. It's less clear whether they can help directly during decode; that depends on whether they can access memory with good throughput and do dequant+compute internally, like GPUs can. Apple Neural Engine only does INT8 or FP16 MADD ops, so that mostly doesn't help.
- avib99 6mo ago[dead]
- philbitt 6mo ago[dead]
- arafeq 6mo ago[flagged]
- metalliqaz 6mo agomy most powerful system is Ryzen+Radeon, so if there are tools that do all the hard work of making AI tools work well on my hardware, I'm all for it. I find it very frustrating to get LLMs, diffusion, etc. working fast on AMD. It's way too much work.
- gnarlouse 6mo agoMaybe it's a language barrier problem, but "by AMD" makes me think its a project distributed by AMD. Is that actually the case? I'm not seeing any reason to believe it is.
- hombre_fatal 6mo ago> You can reach us by filing an issue, emailing lemonade@amd.com Found this on the github readme.
- buildbot 6mo agoIt’s a community project supported and sponsored by AMD according to their GitHub; https://github.com/lemonade-sdk/lemonade https://github.com/lemonade-sdk/lemonade AMD employees work on it/have been making blog posts about it for a bit.
- guipsp 6mo agoIt is mostly developed by AMD and used to be hosted on the AMD github iirc
- AbuAssar 6mo agoCheck the copyright notice at the bottom of the frontage, it says (c) 2026 AMD
- luxuryballs 6mo agothis is funny I’m working on building an AI project called lemonade right now
- pantalaimon 6mo agoIt's pretty annoying that you need vendor specific APIs and a large vendor specific stack to do anything with those NPUs. This way software adoption will be very limited.
- Caum 6mo ago[flagged]
- cl0ckt0wer 6mo agothe npu is more for power efficiency when on battery. I don't think it's a replacement for gpu.
- lrvick 6mo agoI have had way better perf with Vulcan than ROCm on kernel 7.0.0. They made some major improvements. 20%+ speedups for me.
- mindcrime 6mo ago> Been running local LLMs on my 7900 XTX for months and the ROCm experience has been... rough. Just out of curiosity... how so? I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. Actually, I've barely had any problems at all, other than the card being limited to 24GB of VRAM. :-( I'm halfway tempted to splurge on a Radeon Pro board to get more VRAM, but ... haven't bitten the bullet yet.
- skirmish 6mo agoDid you have complete hardware lockups when VRAM is exceeded? I had quite a few on my 7900XTX with llama.cpp (Arch Linux, various driver versions). Once I dial in the quant and context size that never exceed VRAM, it is stable; before that I swear a lot and keep pressing the hardware reset button.
- silask58 6mo ago[dead]
- robotswantdata 6mo agoForget all the vibe coded slop or Ollama. Lemonade is the real deal and very good, been using about a year now. AMD are doing gods work here
- deleted 6mo ago[deleted]
- aplomb1026 6mo ago[dead]
- shubhamgarg86 6mo ago[flagged]
- LowLevelKernel 6mo agoWhich specific NPU’s?
- spencer9714 6mo ago[flagged]
- bravetraveler 6mo agoA fun observation: pulling models sends ~200mbit of progress updates to your browser
- wei03288 6mo ago[dead]
- Sparkyte 6mo agoWhat is the lowest process I can implement this on?
- steffs 6mo agoThe multi-modal bundling is the part that stands out more than the raw inference speed. If you are building an app that needs text generation, image generation, and speech recognition, right now the local setup is three separate services with three different APIs and three different model management stories. Having one server handle all of that behind OpenAI-compatible endpoints is a real quality of life improvement for anyone prototyping locally. The NPU angle is interesting but probably overstated for most use cases. The discussion in the thread confirms what I would expect: NPUs shine for small always-on models and prefill offloading, not for the chatbot workloads most people care about. Where this gets genuinely compelling is if AMD can make the combined GPU plus NPU scheduling transparent enough that developers do not need to think about which hardware is running which part of the pipeline. That is not a solved problem on any platform yet, and if Lemonade gets it right for even a subset of workloads, it becomes the default choice on AMD hardware regardless of how it benchmarks against Ollama on pure text generation.
- aplomb1026 6mo ago[dead]
- devnotes77 6mo ago[dead]
- techpulselab 6mo ago[flagged]
- ozgrakkurt 6mo agoFor people with AMD card. This is garbage, rocm is garbage. Just install llama.cpp and run llama-server with vulkan option. This is just some slop + JS/Electron garbage put on top.
- momo_dev 6mo ago[dead]
- johnwhitman 6mo ago[flagged]
- sparkupcloud 6mo ago[dead]