I managed to get a 4090 hooked up on an Arch machine. It buzzes when the LLM is generating stuff. It isn't a straight forward thing, though, you still have to d

I managed to get a 4090 hooked up on an Arch machine. It buzzes when the LLM is generating stuff. It isn't a straight forward thing, though, you still have to dig through the deps and get your versions just right. On top of that you need to get your versions from bottom-to-top correct for everything to work. Some versions of Cuda don't play nicely with some versions of the Python modules, so take care if you go this route.