5 ms·

A 0.5B parameter model with a 32k context length that also makes good use of that full window?! That's very interesting. The academic benchmarks on that partic

A 0.5B parameter model with a 32k context length that also makes good use of that full window?! That's very interesting. The academic benchmarks on that particular model relative to 1.5B-2B models are what you would expect, but it would make for an excellent base for finetuning/embedding generation.

rgbrgb 2y ago
haven't done anything with a model so small. what kind of stuff would you do with it?
- minimaxir 2y ago
  A properly finetuned model can perform better for a use case but even with PEFT/LoRAs, finetuning and managing "smaller" open-source LLMs (7B params) models like Llama3 is annoying. That's partially why the even-smaller ~2B Phi series of models took off. A 0.5B model may not be that great out of the box but there's a lot of oppertunity if it's responsive to finetuning.
  - danielhanchen 2y ago
    Ye smaller models are fantastic for finetuning and probably on device applications - they can act as like a first pass on most LLM applications, then if it requires a larger model to intervene, then it can pass if off to some larger one. I do have a Colab to finetune 0.5b 2x faster here for those interested: https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6vYeBMHpoePocN?usp=sharing https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6...
- GaggiX 2y ago
  You can use it for speculative decoding for example, this would increase the speed of larger models.
- GuB-42 2y ago
  Tiny LLMs (oxymoron?) could be used for text completion, predictive keyboards, compression, to improve OCR and speech transcription, etc... For these applications, you don't need a super smart model, it just needs to give out hints. For completion, the user can just not use the suggestion if it is not what he wants, for compression, it will just lower the compression ratio a bit, and for transcription, it will only be used as disambiguation. For all these applications, a LLM is not needed, but it can improve the results. Note that for OCR/transcription, I am a bit weary of engines that are a bit too smart, as a lot of them are today, and LLMs go in this direction. The results are often objectively better, that is more words are properly transcribed, but for those that are not, it is often subtly wrong, that is, it makes sense, but it is not the right transcription. Whereas with nonsense, we know it can't be trusted and we act accordingly, also, people can be quite good at filling the blanks themselves. It results in more effort, but maybe better understanding in the end.
3abiton 2y ago
I wonder if the 0.5B model would be usable for ML tasks like summarization, classification, or embeddings, replacing small models like Spacy usually used for embeddings.
- ai_what 2y ago
  It won't. Amazon kind of when that angle with MistralLite[1] (a 7B finetune), and it was barely passing in terms of being an effective summarizer. 0.5B are pretty much useless. https://huggingface.co/amazon/MistralLite https://huggingface.co/amazon/MistralLite
  - coder543 2y ago
    The official Mistral-7B-v0.2 model added support for 32k context, and I think it's far better than MistralLite. Third-party finetunes are rarely amazing at the best of times. Now, we have Mistral-7B-v0.3, which is supposedly an even better model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
  - z4y5f3 2y ago
    My experience is that < 500M models are pretty useful when fine-tuned on traditional NLP tasks, such as text classification and sentence/token level labeling. A modern LM with a 32K context window size could be a nice replacement for BERT, RoBERTa, BART.
GaggiX 2y ago
>that also makes good use of that full window?! To be honest the "Needle In A Haystack" test is the most trivial test for a model that relies on full attention, it's expected to be easy to pass if the model was trained correctly.
- ai_what 2y ago
  I agree. I personally don't have high hopes for the 0.5B model. Phi-2 was 2.7B and it was already regularly outputting complete nonsense. I ran the 0.5B model of the previous Qwen version (1.5) and it reminded me of one of those lorum ipsum word generators. The other new Qwen models (7B and up) look good though.
  - refulgentis 2y ago
    Phi-2 wasn't instruct/chat finetuned and it was very upfront about this, "I tried Phi-2 and it was bad" is a dilletante filter
- falseAss 2y ago
  i just hope people don't claim "X model support Y context window", when the evaluation is done on "Needle in a haystack" only. It creates so much unnecessary hype.
coder543 2y ago
Qwen1.5-0.5B supposedly supported up to 32k context as well, but I can't even get it to summarize a ~2k token input with any level of coherence. I'm always excited to try a new model, so I'm looking forward to trying Qwen2-0.5B... but I wouldn't get your hopes up this much. These super tiny models seem far more experimental than the larger LLMs. Phi-3-mini (3.8B) supports a 128k context, and it is actually a reasonably useful model in my tests. Gemma-2B-1.1-it is a 2B model that only supports 8k context, but it also does fairly well for summarization.
- algo_trader 2y ago
  what would you recommend as a maximum-2B model for coding/reasoning domain ?
  - coder543 2y ago
    For reasoning tasks and coding tasks where you’re chatting with the model, there are no 2B models that I would recommend at this point.
  - selcuka 2y ago
    > maximum-2B model If you are resource limited, remember that you can also play with the quantization to fit more parameters into less amount of RAM. Phi-3-mini [1] (a 3.8B model) is 7.64GB with full (16-bit floating point) precision, but it is only 2.39GB when quantized to 4 bits. That being said, I haven't personally tested it, but have heard good things for CodeGemma 2B [2]. [1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/tree/main https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf... [2] https://huggingface.co/google/codegemma-2b-GGUF https://huggingface.co/google/codegemma-2b-GGUF
    - coder543 2y ago
      CodeGemma-2b does not come in the "-it" (instruction tuned) variant, so it can't be used in a chat context. It is just a base model designed for tab completion of code in an editor, which I agree it is pretty good at.
moffkalast 2y ago
It might be fine tunable for specific tasks BERT-style, I really doubt it's coherent enough for anything out of the box.
danielhanchen 2y ago
For finetuning, I made a Colab notebook to finetune Qwen2 7b 2x faster and use 70% less VRAM than HF+FA2! https://colab.research.google.com/drive/1mvwsIQWDs2EdZxZQF9pRGnnOvE86MVvR?usp=sharing https://colab.research.google.com/drive/1mvwsIQWDs2EdZxZQF9p...

rgbrgb 2y ago

haven't done anything with a model so small. what kind of stuff would you do with it?

minimaxir 2y ago
A properly finetuned model can perform better for a use case but even with PEFT/LoRAs, finetuning and managing "smaller" open-source LLMs (7B params) models like Llama3 is annoying. That's partially why the even-smaller ~2B Phi series of models took off. A 0.5B model may not be that great out of the box but there's a lot of oppertunity if it's responsive to finetuning.
- danielhanchen 2y ago
  Ye smaller models are fantastic for finetuning and probably on device applications - they can act as like a first pass on most LLM applications, then if it requires a larger model to intervene, then it can pass if off to some larger one. I do have a Colab to finetune 0.5b 2x faster here for those interested: https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6vYeBMHpoePocN?usp=sharing https://colab.research.google.com/drive/1-7tjDdMAyeCueyLAwv6...
GaggiX 2y ago
You can use it for speculative decoding for example, this would increase the speed of larger models.
GuB-42 2y ago
Tiny LLMs (oxymoron?) could be used for text completion, predictive keyboards, compression, to improve OCR and speech transcription, etc... For these applications, you don't need a super smart model, it just needs to give out hints. For completion, the user can just not use the suggestion if it is not what he wants, for compression, it will just lower the compression ratio a bit, and for transcription, it will only be used as disambiguation. For all these applications, a LLM is not needed, but it can improve the results. Note that for OCR/transcription, I am a bit weary of engines that are a bit too smart, as a lot of them are today, and LLMs go in this direction. The results are often objectively better, that is more words are properly transcribed, but for those that are not, it is often subtly wrong, that is, it makes sense, but it is not the right transcription. Whereas with nonsense, we know it can't be trusted and we act accordingly, also, people can be quite good at filling the blanks themselves. It results in more effort, but maybe better understanding in the end.

3abiton 2y ago

I wonder if the 0.5B model would be usable for ML tasks like summarization, classification, or embeddings, replacing small models like Spacy usually used for embeddings.

ai_what 2y ago
It won't. Amazon kind of when that angle with MistralLite[1] (a 7B finetune), and it was barely passing in terms of being an effective summarizer. 0.5B are pretty much useless. https://huggingface.co/amazon/MistralLite https://huggingface.co/amazon/MistralLite
- coder543 2y ago
  The official Mistral-7B-v0.2 model added support for 32k context, and I think it's far better than MistralLite. Third-party finetunes are rarely amazing at the best of times. Now, we have Mistral-7B-v0.3, which is supposedly an even better model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
- z4y5f3 2y ago
  My experience is that < 500M models are pretty useful when fine-tuned on traditional NLP tasks, such as text classification and sentence/token level labeling. A modern LM with a 32K context window size could be a nice replacement for BERT, RoBERTa, BART.

GaggiX 2y ago

>that also makes good use of that full window?! To be honest the "Needle In A Haystack" test is the most trivial test for a model that relies on full attention, it's expected to be easy to pass if the model was trained correctly.

ai_what 2y ago
I agree. I personally don't have high hopes for the 0.5B model. Phi-2 was 2.7B and it was already regularly outputting complete nonsense. I ran the 0.5B model of the previous Qwen version (1.5) and it reminded me of one of those lorum ipsum word generators. The other new Qwen models (7B and up) look good though.
- refulgentis 2y ago
  Phi-2 wasn't instruct/chat finetuned and it was very upfront about this, "I tried Phi-2 and it was bad" is a dilletante filter
falseAss 2y ago
i just hope people don't claim "X model support Y context window", when the evaluation is done on "Needle in a haystack" only. It creates so much unnecessary hype.

coder543 2y ago

Qwen1.5-0.5B supposedly supported up to 32k context as well, but I can't even get it to summarize a ~2k token input with any level of coherence. I'm always excited to try a new model, so I'm looking forward to trying Qwen2-0.5B... but I wouldn't get your hopes up this much. These super tiny models seem far more experimental than the larger LLMs. Phi-3-mini (3.8B) supports a 128k context, and it is actually a reasonably useful model in my tests. Gemma-2B-1.1-it is a 2B model that only supports 8k context, but it also does fairly well for summarization.

algo_trader 2y ago
what would you recommend as a maximum-2B model for coding/reasoning domain ?
- coder543 2y ago
  For reasoning tasks and coding tasks where you’re chatting with the model, there are no 2B models that I would recommend at this point.
- selcuka 2y ago
  > maximum-2B model If you are resource limited, remember that you can also play with the quantization to fit more parameters into less amount of RAM. Phi-3-mini [1] (a 3.8B model) is 7.64GB with full (16-bit floating point) precision, but it is only 2.39GB when quantized to 4 bits. That being said, I haven't personally tested it, but have heard good things for CodeGemma 2B [2]. [1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/tree/main https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf... [2] https://huggingface.co/google/codegemma-2b-GGUF https://huggingface.co/google/codegemma-2b-GGUF
  - coder543 2y ago
    CodeGemma-2b does not come in the "-it" (instruction tuned) variant, so it can't be used in a chat context. It is just a base model designed for tab completion of code in an editor, which I agree it is pretty good at.

moffkalast 2y ago

It might be fine tunable for specific tasks BERT-style, I really doubt it's coherent enough for anything out of the box.

danielhanchen 2y ago

For finetuning, I made a Colab notebook to finetune Qwen2 7b 2x faster and use 70% less VRAM than HF+FA2! https://colab.research.google.com/drive/1mvwsIQWDs2EdZxZQF9pRGnnOvE86MVvR?usp=sharing https://colab.research.google.com/drive/1mvwsIQWDs2EdZxZQF9p...