9 ms·
For a great many LLMs, you can find someone on HuggingFace who has produced a set of different quantised versions, with approximate RAM requirements. For examp
by michaelt 2y ago
For a great many LLMs, you can find someone on HuggingFace who has produced a set of different quantised versions, with approximate RAM requirements.
For example, if you want to run "CodeLlama 70B" from https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF where's a table saying the "Q4_K_M" quantised version is a 41.42 GB download and runs in 43.92 GB of memory.