Is this for private/home use? Regardless, I was in a similar situation at the beginning of last year. However, I chose to upgrade my existing PC, which has 64GB

Is this for private/home use? Regardless, I was in a similar situation at the beginning of last year. However, I chose to upgrade my existing PC, which has 64GB RAM and an RTX 3060 TI, with an additional RTX A4000 16G. I bought this used one for about €600 from eBay. Reasoning: This setup is sufficient for the inference of models up to 13B (quantized) and for QLoRA training of the 7B models (4bit q). It is also adequate for experimental training of smaller models like 300M. Thus, this is suitable for tweaking and experimentation on a very small scale. When I need to train larger models, there is always the option of using the cloud. Lambda Labs offers economical prices for this. For training with huge datasets and 34B and larger models, you might want to consider DeepSpeed with ZERO3, which can significantly reduce the cost of training, even with Amazon or Azure instances. Running 70B models will end up being costly. If you will not run this in production mode 24/7 with a profit turnover, then it does not make sense to invest in your own infrastructure.