If you want to wait until the 5090s come out, you should see a drop in the price of the 30xx and 40xx series. Right now, shopping used, you can get two 3090s or

If you want to wait until the 5090s come out, you should see a drop in the price of the 30xx and 40xx series. Right now, shopping used, you can get two 3090s or two 4080s in your price range. Conventional wisdom says two 3090s would be better, but this is all highly dependent on what models you want to run. Basically the first requirement is to have enough VRAM to host all of your model on it, and secondarily, the quality of the GPU. Have a look through Hugging Face to see which models interest you. A rough estimate for the amount of VRAM you need is half the model size plus a couple gigs. So, if using the 70B models interests you, two 4080s wouldn't fit it, but two 3090s would. If you're just interested in the 1B, 3B and 7B models (llama 3B is fantastic), you really don't need much at all. A single 3060 can handle that, and those are not expensive.