4 ms·

I'm curious, what people do with these smaller models?

ignoramous 2y ago
Mistral repeatedly emphasize on "accuracy" and "latency" for this Small (24b) model; which to me means (and as they also point out): - Local virtual assistants. - Local automated workflows. Also from TFA: Our customers are evaluating Mistral Small 3 across multiple industries, including: - Financial services customers for fraud detection - Healthcare providers for customer triaging - Robotics, automotive, and manufacturing companies for on-device command and control - Horizontal use cases across customers include virtual customer service, and sentiment and feedback analysis.
_boffin_ 2y ago
Cleaning messy assessor data. Email draft generation.
frankfrank13 2y ago
They're fast, I used 4o mini to run the final synthesis in a CoT app and to do initial entity/value extraction in an ETL. Mistral is pretty good for code completions too, if I was in the Cursor business I would consider a model like this for small code-block level completions, and let the bigger models handle chat, large requests, etc.
Beretta_Vexee 2y ago
RAG mainly, Feature extraction, tagging, Document and e-mail classification. You don't need a 24B parameter to know whether the e-mail should go to accounting or customer support.
- pheeney 2y ago
  What models would you recommend for basic classification if you don't need a 24B parameter one?
  - josh-sematic 2y ago
    You might find this comparison chart helpful: https://www.airtrain.ai/blog/how-15-top-llms-perform-on-classification-accuracy-vs-cost-breakdown https://www.airtrain.ai/blog/how-15-top-llms-perform-on-clas... Note: from October; also I work at Airtrain
  - Beretta_Vexee 2y ago
    There's no alternative to testing with your own data. The majority of our data is in French, and our benchmarks differ greatly from public benchmarks generally based on English documents.
  - elorant 2y ago
    I’m using Llama-3 8B to classify html files. It’s surprisingly good, and I run it on an RTX 4060 Ti at 8-bit quantization. No complains so far.
- Panoramix 2y ago
  Would this work for non-text data? Like finding outliers in a time series or classifying trends, that kind of thing
  - Beretta_Vexee 2y ago
    Don't use LLM to do something that some python + panda lines could do better. Buy a data scientist a coffee and have a chat.
superkuh 2y ago
Not spend $6000 on hardware because they run on computers we already have. But more seriously, they're fine and plenty fun for making recreational IRC bots.
celestialcheese 2y ago
Classification, tagging tasks. Way easier than older ML techniques and very fast to implement.
- mattgreenrocks 2y ago
  When compared against more traditional ML approaches, how do they fare in terms of quality?
  - spmurrayzzz 2y ago
    Historically the problem with using LLMs for the super simple conventional NLP stuff is that they were hard to control in terms of output. If you wanted a one-word answer for a classification task, you'd often have to deal with it responding in a paragraph. This obviously hurts precision and accuracy quite a bit. There were tricks you could use (like using few-shot examples or GBNF grammars or training low-rank adapters or even re-asking the model) to constrain output a bit, but they weren't perfect. Over the last 12-18 months though, the instruction-following capabilities of the models have improved substantially. This new mistral model in particular is fantastic at doing what you ask. My approach to this personally and professionally is to just benchmark. If I have a classification task, I use a tiny model first, eval both, and see how much improvement I'd get using an LLM. Generally speaking though, the vram costs are so high for the latter that its often not worth it. It really is a case-by-case decision though. Sometimes you want one generic model to do a bunch of tasks rather than train/finetune a dozen small models that you manage in production instead.
  - andrewgross 2y ago
    Super easy to get started, but lacking for larger datasets where you want to understand a bit more about predictions. You generally lose things like prediction probability (though this can be recovered if you chop the head off and just assign output logits to classes instead of tokens), repeatability across experiments, and the ability to tune the model by changing the data. You can still do fine tuning, though itll be more expensive and painfaul than a BERT model. Still, you can go from 0 to ~mostly~ clean data in a few prompts and iterations, vs potentially a few hours with a fine tuning pipeline for BERT. They can actually work well in tandem to bootstrap some training data and then use them together to refine your classification.

Mistral repeatedly emphasize on "accuracy" and "latency" for this Small (24b) model; which to me means (and as they also point out): - Local virtual assistants. - Local automated workflows. Also from TFA: Our customers are evaluating Mistral Small 3 across multiple industries, including: - Financial services customers for fraud detection - Healthcare providers for customer triaging - Robotics, automotive, and manufacturing companies for on-device command and control - Horizontal use cases across customers include virtual customer service, and sentiment and feedback analysis.

_boffin_ 2y ago

Cleaning messy assessor data. Email draft generation.

frankfrank13 2y ago

They're fast, I used 4o mini to run the final synthesis in a CoT app and to do initial entity/value extraction in an ETL. Mistral is pretty good for code completions too, if I was in the Cursor business I would consider a model like this for small code-block level completions, and let the bigger models handle chat, large requests, etc.

Beretta_Vexee 2y ago

RAG mainly, Feature extraction, tagging, Document and e-mail classification. You don't need a 24B parameter to know whether the e-mail should go to accounting or customer support.

pheeney 2y ago
What models would you recommend for basic classification if you don't need a 24B parameter one?
- josh-sematic 2y ago
  You might find this comparison chart helpful: https://www.airtrain.ai/blog/how-15-top-llms-perform-on-classification-accuracy-vs-cost-breakdown https://www.airtrain.ai/blog/how-15-top-llms-perform-on-clas... Note: from October; also I work at Airtrain
- Beretta_Vexee 2y ago
  There's no alternative to testing with your own data. The majority of our data is in French, and our benchmarks differ greatly from public benchmarks generally based on English documents.
- elorant 2y ago
  I’m using Llama-3 8B to classify html files. It’s surprisingly good, and I run it on an RTX 4060 Ti at 8-bit quantization. No complains so far.
Panoramix 2y ago
Would this work for non-text data? Like finding outliers in a time series or classifying trends, that kind of thing
- Beretta_Vexee 2y ago
  Don't use LLM to do something that some python + panda lines could do better. Buy a data scientist a coffee and have a chat.

superkuh 2y ago

Not spend $6000 on hardware because they run on computers we already have. But more seriously, they're fine and plenty fun for making recreational IRC bots.

celestialcheese 2y ago

Classification, tagging tasks. Way easier than older ML techniques and very fast to implement.

mattgreenrocks 2y ago
When compared against more traditional ML approaches, how do they fare in terms of quality?
- spmurrayzzz 2y ago
  Historically the problem with using LLMs for the super simple conventional NLP stuff is that they were hard to control in terms of output. If you wanted a one-word answer for a classification task, you'd often have to deal with it responding in a paragraph. This obviously hurts precision and accuracy quite a bit. There were tricks you could use (like using few-shot examples or GBNF grammars or training low-rank adapters or even re-asking the model) to constrain output a bit, but they weren't perfect. Over the last 12-18 months though, the instruction-following capabilities of the models have improved substantially. This new mistral model in particular is fantastic at doing what you ask. My approach to this personally and professionally is to just benchmark. If I have a classification task, I use a tiny model first, eval both, and see how much improvement I'd get using an LLM. Generally speaking though, the vram costs are so high for the latter that its often not worth it. It really is a case-by-case decision though. Sometimes you want one generic model to do a bunch of tasks rather than train/finetune a dozen small models that you manage in production instead.
- andrewgross 2y ago
  Super easy to get started, but lacking for larger datasets where you want to understand a bit more about predictions. You generally lose things like prediction probability (though this can be recovered if you chop the head off and just assign output logits to classes instead of tokens), repeatability across experiments, and the ability to tune the model by changing the data. You can still do fine tuning, though itll be more expensive and painfaul than a BERT model. Still, you can go from 0 to ~mostly~ clean data in a few prompts and iterations, vs potentially a few hours with a fine tuning pipeline for BERT. They can actually work well in tandem to bootstrap some training data and then use them together to refine your classification.