3 ms·
The short practical answer for inference would be: - You go to https://huggingface.co/TheBloke https://huggingface.co/TheBloke - You select the model you are
by PeterStuer 3y ago
The short practical answer for inference would be:
- You go to https://huggingface.co/TheBloke https://huggingface.co/TheBloke
- You select the model you are interested in
- You check which quantizations would fit your available resources.
- HexDecOctBin 3y agoBut what if you are in the market to buy some hardware? Having some rules of thumb won't hurt.
- PeterStuer 3y agoPersonally I'd go as high as I could afford in this order: 4090 > 3090 > 4080 > 4070 TI > 4070 > 3080 TI > 3080 I have no idea how the 4060 TI 16GB performs Models change from week to week, so just aim for the basics (VRAM, Tensor cores (generation!), and memory bandwidth).