9 ms·
In-memory sounds like the way to go not just in terms of performance, but in that it makes no sense to build an ASIC or program an FPGA for a model that will mo
by moffkalast 2y ago
In-memory sounds like the way to go not just in terms of performance, but in that it makes no sense to build an ASIC or program an FPGA for a model that will most likely be obsolete in a few months at best if you're lucky.
- limit499karma 2y agohttps://arxiv.org/pdf/2402.09709 https://arxiv.org/pdf/2402.09709
- throwawaymaths 2y agoYeah, it's not like foundational models ever share compute kernels, or anything.
- moffkalast 2y agoEh, there's so much shenanigans these days even in fine tuning, people adding empty layers and pruning and whatnot, it's unlikely that even models based on the same one will have the same architecture. For new foundation models it's even worse, because there's some fancy experiment every time and the llama.cpp team needs two weeks to figure out how to implement it so the model can even run.
- throwawaymaths 2y agoYeah so you might need to implement a weird activation or positional enciding in software or something but I suspect 90% will probably be the same... If it's just layer count or skipped matrices I assume it should be possible to write an orchestrator that could run most of those models... Unless we move to mamba or something