4 ms·
If you point it at a huggingface model identifier, it will download it, I assume. No way around that. It reads like it is keeping only the core and the active
by dofm 3mo ago
If you point it at a huggingface model identifier, it will download it, I assume. No way around that.
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
- pvtmert 3mo agoNot an expert in this field, but the "expert" is consisting of multiple layers. To keep it small in terms of memory print, this project streams each layer (dividing even further). It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity. Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed. I assume it's horribly slow, but can be put in a non-intrusive background task...
- dofm 3mo agoThanks for the extra detail here.
- cpfohl 3mo agoObviously needs downloading eventually :). It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.
- jedbrooke 3mo agoI mean, if you had fast enough internet connection, you could just stream it over https