5 ms·
According to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training ex
by nl 3y ago
According to the paper it fine tunes at the speed of inference (!!)
This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.
- moffkalast 3y agoWow if that's true then it's genuinely a complete gamechanger for LLMs as a whole. You probably mean more like 0.3s per token, not per example, but that's still more like 1 or two minutes per training case, not like a day for 4 cases like it is now.
- valval 3y agoI think more importantly, what would the fine tuning routine look like? It's a non-trivial task to dump all of your personal data into any LLM architecture.
- f_devd 3y agoMeZO assumes a smooth parameter space, so you probably won't be able to do it with INT4/8 quantization, probably needs fp8 or smoother.
- isoprophlex 3y agoIf you go through the drudgery of integrating with all the existing channels (mail, Teams, discord, slack, traditional social media, texts, ...), such rapid finetuning speeds could enable an always up to date personality construct, modeled on you. Which is my personal holy grail towards making myself unnecessary; it'd be amazing to be doing some light gardening while the bot handles my coworkers ;)
- deleted 3y ago[deleted]
- vgb2k18 3y ago> while the bot handles my coworkers Or it handles their bots ;)
- gliptic 3y agoI cannot find any such numbers in the paper. What the paper says is that MeZO converges much slower than SGD, and each step needs two forward passes. "As a limitation, MeZO takes many steps in order to achieve strong performance."
- sp332 3y agoIt's the same memory footprint as inference. It's not that fast, and the paper mentions some optimizations that could still be done.
- nl 3y agoYes you are right. I completely misread that!