3 ms·
> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. That's kind of the problem, isn't it
by bastawhiz 3mo ago
> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time.
That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.
- subarctic 3mo agoI know very little about this but it seems like the kind of thing that can and eventually will be solved computationally, not by people figuring out what a parameter or group of parameters represent
- bastawhiz 3mo agoIt's not a compute problem. It's a knowledge problem. Even if you can process each parameter individually and re-link the model layers, you need enough information to know what each parameter is for which is necessarily more memory than the weights themselves. You can use the weights to know whether each parameter is useful for a given prompt, but that operation is a strict superset of just generating the answer. By the time you know which parameters are useful, you've already done all the work of generating your output tokens and the effort is pointless.
- jz391 3mo agoWould an extension to `madvise` to say: "page this region in/out as a whole" help here? Engine could define memory ranges representing each expert and leave paging to the OS (tho' "paging" at this point becomes more similar to swapping in granularity...).
- bastawhiz 3mo agoExperts are chosen per token