4 ms·

haven't done anything with a model so small. what kind of stuff would you do with it?

by rgbrgb 2y ago

minimaxir 2y ago
A properly finetuned model can perform better for a use case but even with PEFT/LoRAs, finetuning and managing "smaller" open-source LLMs (7B params) models like Llama3 is annoying. That's partially why the even-smaller ~2B Phi series of models took off. A 0.5B model may not be that great out of the box but there's a lot of oppertunity if it's responsive to finetuning.
- danielhanchen 2y ago
  Ye smaller models are fantastic for finetuning and probably on device applications - they can act as like a first pass on most LLM applications, then if it requires a larger model to intervene, then it can pass if off to some larger one. I do have a Colab to finetune 0.5b 2x faster here for those interested: https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6vYeBMHpoePocN?usp=sharing https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6...
GaggiX 2y ago
You can use it for speculative decoding for example, this would increase the speed of larger models.
GuB-42 2y ago
Tiny LLMs (oxymoron?) could be used for text completion, predictive keyboards, compression, to improve OCR and speech transcription, etc... For these applications, you don't need a super smart model, it just needs to give out hints. For completion, the user can just not use the suggestion if it is not what he wants, for compression, it will just lower the compression ratio a bit, and for transcription, it will only be used as disambiguation. For all these applications, a LLM is not needed, but it can improve the results. Note that for OCR/transcription, I am a bit weary of engines that are a bit too smart, as a lot of them are today, and LLMs go in this direction. The results are often objectively better, that is more words are properly transcribed, but for those that are not, it is often subtly wrong, that is, it makes sense, but it is not the right transcription. Whereas with nonsense, we know it can't be trusted and we act accordingly, also, people can be quite good at filling the blanks themselves. It results in more effort, but maybe better understanding in the end.

haven't done anything with a model so small. what kind of stuff would you do with it?

by rgbrgb 2y ago

haven't done anything with a model so small. what kind of stuff would you do with it?

A properly finetuned model can perform better for a use case but even with PEFT/LoRAs, finetuning and managing "smaller" open-source LLMs (7B params) models like Llama3 is annoying. That's partially why the even-smaller ~2B Phi series of models took off. A 0.5B model may not be that great out of the box but there's a lot of oppertunity if it's responsive to finetuning.

danielhanchen 2y ago
Ye smaller models are fantastic for finetuning and probably on device applications - they can act as like a first pass on most LLM applications, then if it requires a larger model to intervene, then it can pass if off to some larger one. I do have a Colab to finetune 0.5b 2x faster here for those interested: https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6vYeBMHpoePocN?usp=sharing https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6...

GaggiX 2y ago

You can use it for speculative decoding for example, this would increase the speed of larger models.

GuB-42 2y ago

Tiny LLMs (oxymoron?) could be used for text completion, predictive keyboards, compression, to improve OCR and speech transcription, etc... For these applications, you don't need a super smart model, it just needs to give out hints. For completion, the user can just not use the suggestion if it is not what he wants, for compression, it will just lower the compression ratio a bit, and for transcription, it will only be used as disambiguation. For all these applications, a LLM is not needed, but it can improve the results. Note that for OCR/transcription, I am a bit weary of engines that are a bit too smart, as a lot of them are today, and LLMs go in this direction. The results are often objectively better, that is more words are properly transcribed, but for those that are not, it is often subtly wrong, that is, it makes sense, but it is not the right transcription. Whereas with nonsense, we know it can't be trusted and we act accordingly, also, people can be quite good at filling the blanks themselves. It results in more effort, but maybe better understanding in the end.