3 ms·
Was that implementation confirmed to work or was it a bug? I saw people saying it only loads the weights necessary, but all the weights are necessary (unless t
by gmork13 4y ago
Was that implementation confirmed to work or was it a bug?
I saw people saying it only loads the weights necessary, but all the weights are necessary (unless the architecture is heavily modified). Does it stream or load only the currently necessary layers in a way?
- wsgeorge 4y agoOn start up it does load a fraction of the weights, so RAM usage is low, but as you run inferences on the model, it tends to eat up the required amount of RAM. This was from my initial testing.
- smcameron 4y agoIt mmaps the data so the page any particular bit of data is on is only loaded into memory if something on that page is accessed (at least that is my understanding).
- enlyth 4y agoLooks like it will be reverted because it introduced some regressions and as far as I understand, there was no magic in the end, but there was a ton of drama around it. Most of the drama is contained in this thread, if you like reading this sort of stuff: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment-1493385890 https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...