Tried this awhile ago on the llama models, with my hardware 1. The GPU was still faster, though the difference was smaller than I really expected. 2. Almost a

Tried this awhile ago on the llama models, with my hardware 1. The GPU was still faster, though the difference was smaller than I really expected. 2. Almost all the time was spent on compute, not loading weights to/from the GPU. For models I could fit on my GPU the difference in performance in loading one layer at a time vs loading them all was completely negligible, and the difference in memory usage (constraining what else I could do with the computer simultaneously) significant. 3. Loading from SSD was a huge bottleneck when I tried that Hardware: RTX 2070 super, 5800x