3 ms·
Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40
by syntaxing 14d ago
Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?
- UncleOxidant 14d agoI have not measured pre-fill, but it's said to be around 1000. It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.
- syntaxing 14d agoHalogen as in this right? https://github.com/peonist-ai/halogen-flash-server https://github.com/peonist-ai/halogen-flash-server
- UncleOxidant 14d agoyes
- syntaxing 14d agoThanks! I really like how the author packaged everything into a container. Definitely going to give it a go over the weekend!