> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 LLaMa weights are f16s to start out with, no lowering necessa

> All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 LLaMa weights are f16s to start out with, no lowering necessary to get to there. You can stream weights from RAM to the GPU pretty efficiently. If you have >= 32GB ram and >=2GB vram my code here should work for you: https://github.com/gmorenz/llama/tree/gpu_offload https://github.com/gmorenz/llama/tree/gpu_offload There's probably a cleaner version of it somewhere else. Really you should only need >= 16 GB ram, but the (meta provided) code to load the initial weights is completely unnecessarily making two copies of the weights in RAM simultaneously. You could also lower vram requirements a bit more with a bit more work (I just made the smallest change possible to make it work)