4 ms·
You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.
by petters 3y ago
You don't have to do the loading/discarding explicitly. You could just mmap the entire network and let the os handle that.
- sp332 3y agoDidn't llama.cpp need to convert the weights file to a new format to support that? The way they're stored in the official file isn't efficient for operating on directly.
- LoganDark 3y agoBecause the original format is the undocumented Python pickle format packed into a zip file. It's kind of ridiculous to attempt to support directly.
- gliptic 3y agoThey already had their own format before that.
- petters 3y agoI don't know about llama.cpp, but yes this method works best if the binary layout on disk is exactly what you use for matrices in memory
- samstave 3y ago(I am talking out my butt - because these are new concepts to me, so forgive the ELI5 manner of Qs) ; Can you "peel a 'layer' and feed that off onto somthing that doesnt need to discard, but obly received the "curated" layer via the prompt that drove its creation - and then have other weights assigned? Again - I am infant on this line of questions, so please educate me (the other me myselfs)
- immibis 3y agoThe question is not clear to me, but if you are memory-constrained, you can take a whole batch of inputs, load the first layer into memory, run them through the first layer, unload the first layer, load the second layer, run the first layer outputs through the second layer, and so on.