4 ms·
Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.
by fwip 2mo ago
Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.
- altcognito 2mo agoIt's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson. https://en.wikipedia.org/wiki/Bitter_lesson https://en.wikipedia.org/wiki/Bitter_lesson Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.
- odo1242 2mo agoThe point here is that the library's overhead cost is very low. The fact that a tiny model can reach 10M tokens per second means that the overhead of token decode, memory allocation, calling the model, etc. is very low. The model doesn't actually need to be useful to prove that point.
- voakbasda 2mo agoIt’s interesting and worthy of genuine applaud for being a good starting point for further work. That said, I am more interested in what size model this could manage while producing “just enough” tokens per second to work at a conversational rate. What are models in that class capable of doing for me?
- altcognito 2mo agoThank you for your kind reply. I appreciate your point completely and while I tried to moderate sounding dismissive of what was being done here, I think I could have done better. I love "trivial" examples and everything you've said is tue.
- entrope 2mo agoIs it that impressive? It is a model generating short strings from scratch, so I do not think there is significant token parsing or memory allocation going on. In compute performance, 10M tok/s * 4096 parameter/tok * 4 byte/param = ~160 GiB/sec implied memory bandwidth, vs nominal ~300 GB/sec for the M5 Pro's RAM. That seems even less notable once we consider that 4096 parameter * 4 byte/param = 16 KiB of parameters fit easily within L1 data cache (the M5's efficiency cores each have 64 KiB of L1 data cache). 40 GFLOPs means ~10 FLOP/cycle, about 2.5 NEON instructions per cycle given that the core instructions are fused multiply-adds (FMAs). A random web page I found says the M5 family cores have a 4-wide SIMD block, meaning this gets about 63% utilization: respectable but not super high.
- alightsoul 2mo agoI think the bitter lesson only talks about task performance but not computational efficiency. Could tiny models improve efficiency? Maybe by just using a general architecture on specialized data, so the artichecture itself is not task specific?
- mcbuilder 2mo agoSmall # of parameters means no memory bottleneck, which means blazing fast performance.
- andai 2mo agoWait, does this fit in L1?
- fwip 2mo agoL1 cache for a M5 is like 128KB, right? So I can't see how it wouldn't.