6 ms·
LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows
- xnx 3y agoThese types of optimizations are what I figured Carmack would be doing at Keen. He's so good at optimization and low-level hardware understanding that I thought he might be able to wring some small extra percentage out of existing hardware for AI tasks. With compute currently as expensive as it is, that would be extremely valuable.
- seaal 3y agoLatest 545.84 released 10.17 NVIDIA drivers have impressive performance improvements.
- monkeydust 3y agoNoticeable for gaming on a 3090? Working my way through Phantom Liberty DLC and massively enjoying it.
- gentleman11 3y agoIn other news: GeForce experience continues to require an account and has creepy privacy issues
- ramon156 3y agosorry bud, we cant make your games super duper fast if you dont give us your email!
- gattilorenz 3y agoemail? They’re sweeping up an incredible amount of data, basically every window you open.
- garblegarble 3y agoDo you have more info on that? Their support site[1] claims their data collection is much more constrained than that (although still seems like a lot). 1: https://nvidia.custhelp.com/app/answers/detail/a_id/3188/~/what-data-does-geforce-experience-send-to-nvidia%3F https://nvidia.custhelp.com/app/answers/detail/a_id/3188/~/w...
- gattilorenz 3y agoAh, you’re right! I’m basically repeating stuff I got from a somewhat recent HN thread [1] which I found shocking when I first read it, except it’s not true and I did not bother checking it because it was not unthinkable and… confirmation bias, I guess. Now, if only I could go back and still edit the original post… Thanks for catching my bullshit, I stand corrected (and should have been wiser). [1] https://news.ycombinator.com/item?id=37033859 https://news.ycombinator.com/item?id=37033859
- woleium 3y agoThis is the HN I love :)
- garblegarble 3y agoNo worries, confirmation bias comes for us all at some point!
- murermader 3y agoIt is less convenient (by design), but you can always manually look for new drivers and download them. Thats how I do it. Maybe someone wrote a script that automatically scrapes the website and downloads the newest driver automatically?
- 1696343940 3y agoTechPowerUp's nvcleanstall is a pretty lean NVidia driver installer. It fetches from TPUs servers, and installs only user-selected driver components. The screenshots linked do a good job of communicating what it does. Has a scheduled update-check if you want that too. https://www.techpowerup.com/nvcleanstall/ https://www.techpowerup.com/nvcleanstall/
- SillyUsername 3y agoI'm looking forward to CodeLlama being usable on a Dev system using some CPU offloading and only 32GB _while developing in an IDE_. Code Llama is great but using it alongside an IDE just kicks swapping into overdrive, and at least this will mitigate some of that loss on the LLM generation side (at least for the part in GPU ram).
- vlovich123 3y agoDo you use https://github.com/xNul/code-llama-for-vscode https://github.com/xNul/code-llama-for-vscode or something else? Haven’t found any good setup instructions for Linux or my Google skills are failing me.
- SillyUsername 3y agoI use https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui separately as I code in VS 2022, VS Code, Intellij Idea and Webstorm
- bavell 3y agoI just setup vanilla llama.cpp this past weekend - good place to start as an intro IMO
- FirmwareBurner 3y agoWhere is AMD in all this? Like M1 Macs, their Ryzen APUs have access to the whole system memory and AI accelerators, so I would like to get a laptop with 32-64GB of RAM just to fit giant LLMs in it, instead of splurging a lot more on RTX cards with large VRAM even if I don't game, but AMD isn't even trying.
- T-A 3y agohttps://www.reuters.com/markets/deals/amd-acquire-ai-software-startup-effort-catch-nvidia-2023-10-10/ https://www.reuters.com/markets/deals/amd-acquire-ai-softwar...
- FirmwareBurner 3y agoSure, but Apple laptops can run Llama right now. How many years do we have to wait till AMDs aquisition efforts materialize in products we can buy?
- kramerger 3y agoI dont understand, are you claiming non-Apple devices cannot run LLMs? https://github.com/ggerganov/llama.cpp/issues/34 https://github.com/ggerganov/llama.cpp/issues/34 If you meant eGPU support, IIRC that is beta for everyone right now.
- KeplerBoy 3y agoDDR4 and DDR5 are nowhere near as fast the memory in Apple's SoCs or GDDR6X found on modern GPUs.
- FirmwareBurner 3y agoBut AMD can in theory also use the same DRR that Apple uses. There ale laptops already with LPDDR5X-8000 frequency.
- IshKebab 3y agoReally? Wikipedia says DDR5 supports 8 GT/s and GDDR6X supports 16. I mean I guess they go a lot wider on a GPU, but there's no reason an APU couldn't do that too.
- 71a54xd 3y agoI assume they released equivalents for Linux?
- OMGnotThatGuy 3y agoIt appears that it's already in there. The trick is that you have to install an extension in some apps to make it work. For instance, there's a TensorRT accelerator extension for Stable Diffusion WebUI that takes advantage of it. I'm installing it right now, though the docs leave a bit to be desired. https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT
- OMGnotThatGuy 3y agoI got it installed and tested with an SD 1.5 model in Stable Diffusion Webui using a 4090. (SDXL models did not work.) I generated the same test prompt with and without their TensorRT acceleration. The generated images are nearly identical, with very minor, almost imperceptible differences, which is pretty standard for techniques that optimize the attention layers. Prompt: A cat wearing pajamas Model: deliberate_v2 Size: 768x512 Batch Size: 4 Clip Skip: 2 Samler: DPM++ 2S a Karras Steps: 70 Seed: 2024828515 Generation times avg over 5 runs ================================ without TensorRT: 19.2 seconds with TensorRT: 12.3 seconds Benchmarks Batch Size 1 / 2 / 4 ======================================= without TensorRT: 31.11 / 36.23 / 42.34 with TensorRT: 55.32 / 58.06 / 62.27 It's faster, but clearly it's not 4x faster. I suppose they cherrypicked benchmarks against generation techniques not using xformers or SDP Attention. Also, it appears to be limited to a max batch size of 4.
- gbertb 3y agowas wondering about this
- daft_pink 3y agodoes this include wsl or just regular windows?
- holbrad 3y agoDoes fix the regression that was causing poor stable diffusion inference ?
- looppp 3y ago[dead]