4 ms·

May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks suc

May I ask if anyone has successfully used 1B and 3B models in production and if yes, in what use cases? I seem to be failing even in seemingly simpler tasks such as word translation or zero-shot classification. For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline :/

wswope 2y ago
I’ve only toyed with them a bit, and had a similar experience - but did find I got better output by forcing them to adhere to a fixed grammar: https://github.com/ggerganov/llama.cpp/tree/master/grammars https://github.com/ggerganov/llama.cpp/tree/master/grammars For context, I was playing with a script to bulk download podcasts, transcribe with whisper, pass the transcription to llama.cpp to ID ads, then slice the ads out with ffmpeg. I started with the generic json_array example grammar, then iteratively tweaked it.
accrual 2y ago
Not in production, but I've used a 3B model to test a local LLM application I'm working on. I needed a full end-to-end request/response and it's a lot faster asking a 3B model than an 8B model. I could setup a test harness and replay the responses... but this was a lot simpler.
- jdthedisciple 2y ago
  If for testing then why not just mock the whole thing for ultimate performance ... ?
  - nkozyra 2y ago
    Probably faster to use off the shelf model with llama.cpp than to mock it
com2kid 2y ago
3B models are perfectly capable, I've had great luck with Phi 3.5. > For example, they seem to not care about instructions to only write a response and no explanation You need to use tools to force the model to adhere to a schema. Or you can learn to parse out the part of the response you want, both work. You'll also need to make good use of robust examples in your initial prompt, and give lots of examples of how you want the output to look. (Yes this quickly burns up the limited context length!) Finally, embrace the fact that these models are tuned for chat, so the more conversational you make the back and forth the less you are stretching the models abilities. I wrote a very small blog post at https://meanderingthoughts.hashnode.dev/unlock-the-full-potential-of-llms-ditch-json-for-dsls https://meanderingthoughts.hashnode.dev/unlock-the-full-pote... explaining some of this.
- teleforce 2y ago
  I wonder if CUE can help the situation in similar fashion to the DSL methods that you've described in your blog post [1]. After all CUE fundamentals are based on feature structure from the deterministic approach of NLP unlike LLM that's stochastic NLP [2],[3]. Perhaps deterministic and non-deterministic approaches is the potent combination that can effectively help reduce much of the footprint to get to the same results and being energy efficient in the process. [1] Cue – A language for defining, generating, and validating data: https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943 [2] Feature structure: https://en.m.wikipedia.org/wiki/Feature_structure https://en.m.wikipedia.org/wiki/Feature_structure [3] The Logic of CUE: https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/
  - com2kid 2y ago
    On my LinkedIn post about this topic someone actually replied with a superior method of steering LLM output compared to anything else I've ever heard of, so I've decided that until I find time to implement their method, I'm not going to worry about things. tl;dr you put into the prompt all the JSON up until what you want the LLM to say, and you set the stop token to the end token of the current JSON item (so ',' or '}' ']', whatever) and you then your code fills out the rest of the JSON syntax up until another LLM generated value is needed. I hope that makes sense. It is super cool, and I am pretty sure there is a way to make a generator that takes in an arbitrary JSON schema and builds a state machine to do the above. The performance should be super fast on locally hosted models that are using context caching. Eh I should write this up as a blog post, hope someone else implements it, and if not, just do it myself.
JohnHammersley 2y ago
> For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline I was doing some local tidying up of recording transcripts, using a fairly long system prompt, and I saw the same behaviour you mention if the transcript I was passing in was too long -- batching it up to make sure to be under the max length prevented this. Might not be what's happening in your case, but I mention it because it wasn't immediately obvious to me when I first saw the behaviour.
beoberha 2y ago
For me, it was almost random if I would get a little spiel at the beginning of my response - even on the unquantized 8b instruct. Since ollama doesn’t support grammars, I was trying to get it to work where I had a prompt that summarized an article and extracted and classified certain information that I requested. Then I had another prompt that would digest the summary and spit out a structured JSON output. It was much better than trying to do it in one prompt, but still far too random even with temperature at 0. Sometimes the first prompt misclassified things. Sometimes the second prompt would include a “here’s your structured output”. And Claude did everything perfectly ;)
- BoorishBears 2y ago
  Why not preprompt with ```json {
  - jkukul 2y ago
    Yes, you can pre-fill the assistant's response with "```json {" or even "{" and that should increase the likelihood of getting a proper JSON in the response, but it's still not guaranteed. It's not nearly reliable enough for a production use case, even on a bigger (8B) model. I could recommend using ollama or VLLm inference servers. They support a `response_format="json"` parameter (by implementing grammars on top of the base model). It makes it reliable for a production use, but in my experience the quality of the response decreases slightly when a grammar is applied.
    - BoorishBears 2y ago
      Grammars are best but if you read their comment they're apparently using ollama in a situation that doesn't support them.
scriptsmith 2y ago
Yes, I've used the v3.2 3B-Instruct model in a Slack app. Specifically using vLLM, with a template: https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_llama3.2_json.jinja https://github.com/vllm-project/vllm/blob/main/examples/tool... Works as expected if you provide a few system prompts with context.
bloomingkales 2y ago
Qwen2.5 3b is very very good.
nikolayasdf123 2y ago
+1 1B and 3B models perform so poorly, it is bellow any acceptance for us. and we have fairly simple natural language understanding.
ipsum2 2y ago
You can't expect a 1B model to perform as well as 7B or chatGPT, probably the best use case is speculative decoding or to use to fine tune for a specific use case.
- theanonymousone 2y ago
  What is "speculative decoding"?
  - regularfry 2y ago
    Speculative decoding is using a small model to quickly generate a sequence that every so often you pass through a larger model to check and correct. It can be much faster than just using the larger model, with tolerably close accuracy.
    - qeternity 2y ago
      > with tolerably close accuracy. No, speculative decoding has exactly the same accuracy as the target model. It is mathematically identical to greedy decoding.
      - kgc 2y ago
        Is there a reference for this? I was wondering the same thing.
        qeternity 2y ago
        Read the original whitepaper or go look at how any framework implements it. You will see that tokens not predicted by greedy sampling of the target model are rejected. Ergo, they are mathematically identical.
blinkingled 2y ago
Just tried asking Llama 3.2:3b to write a YAML file with Kubernetes Deployment definition. It spit the yaml out but along with a ton of explanations. But when I followed up with below it did what I want it to do. >>> Remove the explanation parts and only leave yaml in place from above response. apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment spec: replicas: 3 ... Alternatively this worked as well >>> Write a YAML file with kubernetes deployment object in it. Response should only contain the yaml file, no explanations. ... ions. ```yml apiVersion: apps/v1 kind: Deployment metadata: name: example-deployment spec: replicas: 3 selector: matchLabels: app: example-app template: metadata: labels: app: example-app spec: containers: - name: example-container image: nginx:latest ports: - containerPort: 80 ```
itsTyrion 2y ago
I've tried using 3B outside of production. Asked it to be the character needed, like 30 words and use German. Instructions were consistently ignored, sometimes sentences devolved into Gibberish or English was mixed in halfway through. Don't even want to know how lobotomized 1B is.

wswope 2y ago

I’ve only toyed with them a bit, and had a similar experience - but did find I got better output by forcing them to adhere to a fixed grammar: https://github.com/ggerganov/llama.cpp/tree/master/grammars https://github.com/ggerganov/llama.cpp/tree/master/grammars For context, I was playing with a script to bulk download podcasts, transcribe with whisper, pass the transcription to llama.cpp to ID ads, then slice the ads out with ffmpeg. I started with the generic json_array example grammar, then iteratively tweaked it.

accrual 2y ago

Not in production, but I've used a 3B model to test a local LLM application I'm working on. I needed a full end-to-end request/response and it's a lot faster asking a 3B model than an 8B model. I could setup a test harness and replay the responses... but this was a lot simpler.

jdthedisciple 2y ago
If for testing then why not just mock the whole thing for ultimate performance ... ?
- nkozyra 2y ago
  Probably faster to use off the shelf model with llama.cpp than to mock it

com2kid 2y ago

3B models are perfectly capable, I've had great luck with Phi 3.5. > For example, they seem to not care about instructions to only write a response and no explanation You need to use tools to force the model to adhere to a schema. Or you can learn to parse out the part of the response you want, both work. You'll also need to make good use of robust examples in your initial prompt, and give lots of examples of how you want the output to look. (Yes this quickly burns up the limited context length!) Finally, embrace the fact that these models are tuned for chat, so the more conversational you make the back and forth the less you are stretching the models abilities. I wrote a very small blog post at https://meanderingthoughts.hashnode.dev/unlock-the-full-potential-of-llms-ditch-json-for-dsls https://meanderingthoughts.hashnode.dev/unlock-the-full-pote... explaining some of this.

teleforce 2y ago
I wonder if CUE can help the situation in similar fashion to the DSL methods that you've described in your blog post [1]. After all CUE fundamentals are based on feature structure from the deterministic approach of NLP unlike LLM that's stochastic NLP [2],[3]. Perhaps deterministic and non-deterministic approaches is the potent combination that can effectively help reduce much of the footprint to get to the same results and being energy efficient in the process. [1] Cue – A language for defining, generating, and validating data: https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943 [2] Feature structure: https://en.m.wikipedia.org/wiki/Feature_structure https://en.m.wikipedia.org/wiki/Feature_structure [3] The Logic of CUE: https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/
- com2kid 2y ago
  On my LinkedIn post about this topic someone actually replied with a superior method of steering LLM output compared to anything else I've ever heard of, so I've decided that until I find time to implement their method, I'm not going to worry about things. tl;dr you put into the prompt all the JSON up until what you want the LLM to say, and you set the stop token to the end token of the current JSON item (so ',' or '}' ']', whatever) and you then your code fills out the rest of the JSON syntax up until another LLM generated value is needed. I hope that makes sense. It is super cool, and I am pretty sure there is a way to make a generator that takes in an arbitrary JSON schema and builds a state machine to do the above. The performance should be super fast on locally hosted models that are using context caching. Eh I should write this up as a blog post, hope someone else implements it, and if not, just do it myself.

JohnHammersley 2y ago

> For example, they seem to not care about instructions to only write a response and no explanation, thus making it impossible to use them in a pipeline I was doing some local tidying up of recording transcripts, using a fairly long system prompt, and I saw the same behaviour you mention if the transcript I was passing in was too long -- batching it up to make sure to be under the max length prevented this. Might not be what's happening in your case, but I mention it because it wasn't immediately obvious to me when I first saw the behaviour.

beoberha 2y ago

For me, it was almost random if I would get a little spiel at the beginning of my response - even on the unquantized 8b instruct. Since ollama doesn’t support grammars, I was trying to get it to work where I had a prompt that summarized an article and extracted and classified certain information that I requested. Then I had another prompt that would digest the summary and spit out a structured JSON output. It was much better than trying to do it in one prompt, but still far too random even with temperature at 0. Sometimes the first prompt misclassified things. Sometimes the second prompt would include a “here’s your structured output”. And Claude did everything perfectly ;)

BoorishBears 2y ago
Why not preprompt with ```json {
- jkukul 2y ago
  Yes, you can pre-fill the assistant's response with "```json {" or even "{" and that should increase the likelihood of getting a proper JSON in the response, but it's still not guaranteed. It's not nearly reliable enough for a production use case, even on a bigger (8B) model. I could recommend using ollama or VLLm inference servers. They support a `response_format="json"` parameter (by implementing grammars on top of the base model). It makes it reliable for a production use, but in my experience the quality of the response decreases slightly when a grammar is applied.
  - BoorishBears 2y ago
    Grammars are best but if you read their comment they're apparently using ollama in a situation that doesn't support them.

scriptsmith 2y ago

Yes, I've used the v3.2 3B-Instruct model in a Slack app. Specifically using vLLM, with a template: https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_llama3.2_json.jinja https://github.com/vllm-project/vllm/blob/main/examples/tool... Works as expected if you provide a few system prompts with context.

bloomingkales 2y ago

Qwen2.5 3b is very very good.

nikolayasdf123 2y ago

+1 1B and 3B models perform so poorly, it is bellow any acceptance for us. and we have fairly simple natural language understanding.

ipsum2 2y ago

You can't expect a 1B model to perform as well as 7B or chatGPT, probably the best use case is speculative decoding or to use to fine tune for a specific use case.

theanonymousone 2y ago
What is "speculative decoding"?
- regularfry 2y ago
  Speculative decoding is using a small model to quickly generate a sequence that every so often you pass through a larger model to check and correct. It can be much faster than just using the larger model, with tolerably close accuracy.
  - qeternity 2y ago
    > with tolerably close accuracy. No, speculative decoding has exactly the same accuracy as the target model. It is mathematically identical to greedy decoding.
    - kgc 2y ago
      Is there a reference for this? I was wondering the same thing.
      - qeternity 2y ago
        Read the original whitepaper or go look at how any framework implements it. You will see that tokens not predicted by greedy sampling of the target model are rejected. Ergo, they are mathematically identical.

blinkingled 2y ago

Just tried asking Llama 3.2:3b to write a YAML file with Kubernetes Deployment definition. It spit the yaml out but along with a ton of explanations. But when I followed up with below it did what I want it to do. >>> Remove the explanation parts and only leave yaml in place from above response. apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment spec: replicas: 3 ... Alternatively this worked as well >>> Write a YAML file with kubernetes deployment object in it. Response should only contain the yaml file, no explanations. ... ions. ```yml apiVersion: apps/v1 kind: Deployment metadata: name: example-deployment spec: replicas: 3 selector: matchLabels: app: example-app template: metadata: labels: app: example-app spec: containers: - name: example-container image: nginx:latest ports: - containerPort: 80 ```

itsTyrion 2y ago

I've tried using 3B outside of production. Asked it to be the character needed, like 30 words and use German. Instructions were consistently ignored, sometimes sentences devolved into Gibberish or English was mixed in halfway through. Don't even want to know how lobotomized 1B is.