4 ms·

Qwen1.5-0.5B supposedly supported up to 32k context as well, but I can't even get it to summarize a ~2k token input with any level of coherence. I'm always exc

Qwen1.5-0.5B supposedly supported up to 32k context as well, but I can't even get it to summarize a ~2k token input with any level of coherence. I'm always excited to try a new model, so I'm looking forward to trying Qwen2-0.5B... but I wouldn't get your hopes up this much. These super tiny models seem far more experimental than the larger LLMs. Phi-3-mini (3.8B) supports a 128k context, and it is actually a reasonably useful model in my tests. Gemma-2B-1.1-it is a 2B model that only supports 8k context, but it also does fairly well for summarization.

algo_trader 2y ago
what would you recommend as a maximum-2B model for coding/reasoning domain ?
- coder543 2y ago
  For reasoning tasks and coding tasks where you’re chatting with the model, there are no 2B models that I would recommend at this point.
- selcuka 2y ago
  > maximum-2B model If you are resource limited, remember that you can also play with the quantization to fit more parameters into less amount of RAM. Phi-3-mini [1] (a 3.8B model) is 7.64GB with full (16-bit floating point) precision, but it is only 2.39GB when quantized to 4 bits. That being said, I haven't personally tested it, but have heard good things for CodeGemma 2B [2]. [1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/tree/main https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf... [2] https://huggingface.co/google/codegemma-2b-GGUF https://huggingface.co/google/codegemma-2b-GGUF
  - coder543 2y ago
    CodeGemma-2b does not come in the "-it" (instruction tuned) variant, so it can't be used in a chat context. It is just a base model designed for tab completion of code in an editor, which I agree it is pretty good at.
noman-land 2y ago
Phi-3-mini has been really surprising me. It's quite good!
sitkack 2y ago
Summarization is one of the most difficult tasks for any LLM and over that context window, crazy to think it could do it. That context window is useful if you have a smaller data extraction task, like dates, times, place names, etc. And even that it might need to be fine tuned on. These small models are a feedstock.
TechDebtDevin 2y ago
What tasks do you consider a 3.8B model to be useful for? Chat applications on lesser hardware, im still finding it difficult to parse what the real world application would ever be. However, I do understand that the goal is to make the smallest most efficient model to compete with the larger model capabilities one day and you can't get there without making these. But do these types of models have any value for any sort of product or real world project?
- coder543 2y ago
  I think most of the interesting applications for these small models are in the form of developer-driven automations, not chat interfaces. A common example that keeps popping up is a voice recorder app that can provide not just a transcription of the recording (which you don't need an LLM for), but also a summary of the transcription, including key topics, key findings, and action items that were discussed in a meeting. With speaker diarization (assigning portions of the transcript to different speakers automatically), it's even possible to use an LLM to assign names to each of the speakers in the transcript, if they ever identified themselves in the meeting, and then the LLM could take that and also know who is supposed to be handling each action item, if that was discussed in the meeting. That's just scratching the surface of what should be possible using small LLMs (or SLMs, as Microsoft likes to call them). An on-device LLM could summarize notifications if you have a lot of catching up to do, or it could create a title for a note automatically once you finish typing the note, or it could be used to automatically suggest tags/categories for notes. That LLM could be used to provide "completions", like if the user is writing a list of things in a note, the user could click a button to have that LLM generate several more items following the same theme. That LLM can be used to suggest contextually-relevant quick replies for conversations. In a tightly-integrated system, you could imagine receiving a work phone call, and that LLM could automatically summarize your recent interactions with that person (across sms, email, calendar, and slack/teams) for you on the call screen, which could remind you why they're calling you. LLMs can also be used for data extraction, where they can be given unstructured text, and fill in a data structure with the desired values. As an example, one could imagine browsing a job posting... the browser could use an LLM to detect that the primary purpose of this webpage is a job posting, and then it could pass the text of the page through the LLM and ask the LLM to fill in common values like the job title, company name, salary range, and job requirements, and then the browser could offer a condensed interface with this information, as well as the option to save this information (along with the URL to the job posting) to your "job search" board with one click. Now, it might be a little much to ask a browser to have special cases for just job postings, when there are so many similar things a user might want to save for later, so you could even let the user define new "boards" where they describe to a (hopefully larger) LLM the purpose of the board and the kinds of information you're looking for, and it would generate the search parameters and data extraction tasks that a smaller LLM would then do in the background as you browse, letting the browser present that information when it is available so that you can choose whether to save it to your board. The larger LLM could still potentially be on-device, but a more powerful LLM that occupies most of the RAM and processing on your device is something you'd only want to use for a foreground task, not eating up resources in the background. LLMs are interesting because they make it possible to do things that traditional programming could not do in any practical sense. If something can be done without an LLM, then absolutely... do that. LLMs are very computationally intensive, and their accuracy is more like a human than a computer. There are plenty of drawbacks to LLMs, if you have another valid option.

algo_trader 2y ago

what would you recommend as a maximum-2B model for coding/reasoning domain ?

coder543 2y ago
For reasoning tasks and coding tasks where you’re chatting with the model, there are no 2B models that I would recommend at this point.
selcuka 2y ago
> maximum-2B model If you are resource limited, remember that you can also play with the quantization to fit more parameters into less amount of RAM. Phi-3-mini [1] (a 3.8B model) is 7.64GB with full (16-bit floating point) precision, but it is only 2.39GB when quantized to 4 bits. That being said, I haven't personally tested it, but have heard good things for CodeGemma 2B [2]. [1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/tree/main https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf... [2] https://huggingface.co/google/codegemma-2b-GGUF https://huggingface.co/google/codegemma-2b-GGUF
- coder543 2y ago
  CodeGemma-2b does not come in the "-it" (instruction tuned) variant, so it can't be used in a chat context. It is just a base model designed for tab completion of code in an editor, which I agree it is pretty good at.

noman-land 2y ago

Phi-3-mini has been really surprising me. It's quite good!

sitkack 2y ago

Summarization is one of the most difficult tasks for any LLM and over that context window, crazy to think it could do it. That context window is useful if you have a smaller data extraction task, like dates, times, place names, etc. And even that it might need to be fine tuned on. These small models are a feedstock.

TechDebtDevin 2y ago

What tasks do you consider a 3.8B model to be useful for? Chat applications on lesser hardware, im still finding it difficult to parse what the real world application would ever be. However, I do understand that the goal is to make the smallest most efficient model to compete with the larger model capabilities one day and you can't get there without making these. But do these types of models have any value for any sort of product or real world project?

coder543 2y ago
I think most of the interesting applications for these small models are in the form of developer-driven automations, not chat interfaces. A common example that keeps popping up is a voice recorder app that can provide not just a transcription of the recording (which you don't need an LLM for), but also a summary of the transcription, including key topics, key findings, and action items that were discussed in a meeting. With speaker diarization (assigning portions of the transcript to different speakers automatically), it's even possible to use an LLM to assign names to each of the speakers in the transcript, if they ever identified themselves in the meeting, and then the LLM could take that and also know who is supposed to be handling each action item, if that was discussed in the meeting. That's just scratching the surface of what should be possible using small LLMs (or SLMs, as Microsoft likes to call them). An on-device LLM could summarize notifications if you have a lot of catching up to do, or it could create a title for a note automatically once you finish typing the note, or it could be used to automatically suggest tags/categories for notes. That LLM could be used to provide "completions", like if the user is writing a list of things in a note, the user could click a button to have that LLM generate several more items following the same theme. That LLM can be used to suggest contextually-relevant quick replies for conversations. In a tightly-integrated system, you could imagine receiving a work phone call, and that LLM could automatically summarize your recent interactions with that person (across sms, email, calendar, and slack/teams) for you on the call screen, which could remind you why they're calling you. LLMs can also be used for data extraction, where they can be given unstructured text, and fill in a data structure with the desired values. As an example, one could imagine browsing a job posting... the browser could use an LLM to detect that the primary purpose of this webpage is a job posting, and then it could pass the text of the page through the LLM and ask the LLM to fill in common values like the job title, company name, salary range, and job requirements, and then the browser could offer a condensed interface with this information, as well as the option to save this information (along with the URL to the job posting) to your "job search" board with one click. Now, it might be a little much to ask a browser to have special cases for just job postings, when there are so many similar things a user might want to save for later, so you could even let the user define new "boards" where they describe to a (hopefully larger) LLM the purpose of the board and the kinds of information you're looking for, and it would generate the search parameters and data extraction tasks that a smaller LLM would then do in the background as you browse, letting the browser present that information when it is available so that you can choose whether to save it to your board. The larger LLM could still potentially be on-device, but a more powerful LLM that occupies most of the RAM and processing on your device is something you'd only want to use for a foreground task, not eating up resources in the background. LLMs are interesting because they make it possible to do things that traditional programming could not do in any practical sense. If something can be done without an LLM, then absolutely... do that. LLMs are very computationally intensive, and their accuracy is more like a human than a computer. There are plenty of drawbacks to LLMs, if you have another valid option.