> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests

> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference). The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized. For local usage, I'd say you will have an "OK" experience in a recent laptop with 64GB RAM, which is more realistic than having the necessary VRAM in a Nvidia GPU.