4 ms·

What Mistral has though is speed, and with speed comes scale.

spaceman_2020 3y ago
Who cares about speed if you’re wrong? This isn’t a race to write the most lines of code or the most lines of text. It’s a race to write the most correct lines of code. I’ll wait half an hour for a response if I know I’m getting at least staff engineer level tier of code for every question
- popinman322 3y ago
  For the tasks my group is considering, even a 7B model is adequate. Sufficiently accurate responses can be fed into other systems downstream and cleaned up. Even code responses can benefit from this by restricting output tokens using the grammar of the target language, or iterating until the code compiles successfully. And for a decent number of LLM-enabled use cases the functionality unlocked by these models is novel. When you're going from 0 to 1 people will just be amazed that the product exists.
- sjwhevvvvvsj 3y ago
  Who says it’s wrong? I have very discrete tasks which involve resolving linguistic ambiguity and they can perform very well.
  - mlnj 3y ago
    Exactly. Not everything is throwing large chunks of text to get complex questions answered. I love using the smaller models like Starling LM 7B and Mistral 7B have been enough for many tasks like you mentioned.
- ein0p 3y ago
  That’s the correct answer. Years ago I worked on inference efficiency on edge hardware at a startup. Time after time I saw that users vastly prefer slower, but more accurate and robust systems. Put succinctly: nobody cares how quick a model is if it doesn’t do a good job. Another thing I discovered is it can be very difficult to convince software engineers of this obvious fact.
  - spacecadet 3y ago
    Having spent time on edge compute projects. This. Also, all the evidence is in this thread. Clearly people unhappy with wasting time on LLMs, when the time that was wasted was the result of obviously bad output.
    - sjwhevvvvvsj 3y ago
      People think LLM are all or nothing, like it’s either god-like AGI or it’s useless “hallucinating”. In reality you have to know the strengths and weaknesses of any tool, and small/fast LLM can do a tremendous amount within a fixed scope. The people at Mistral get this.
  - Al-Khwarizmi 3y ago
    Less compute also means lower cost, though. I see how most people would prefer a better but slower model when price is equal, but I'm sure many prefer a worse $2/mo model over a better $20/mo model.
    - ein0p 3y ago
      That’s the thing I’m finding so hard to explain. Nobody would ever pay even $2 for a system that is worse at solving the problem. There is some baseline compute you need to deliver certain types of models. Going below that level for lower cost at the expense of accuracy and robustness is a fool’s errand. In LLMs it’s even worse. To make it concrete, for how I use LLMs I will not only not pay for anything with less capability than GPT4, I won’t even use it for free. It could be that other LLMs could perform well on narrow problems after fine tuning, but even then I’d prefer the model with the highest metrics, not the lowest inference cost.
      - sjwhevvvvvsj 3y ago
        So I think that’s a “your problem isn’t right for the tool” issue, not a “Mistral isn’t capable” issue.
- dathinab 3y ago
  Who care about getting better answers if you can't afford it, can't use it for legal reason or conclude that the risk associated with OpenAI now being a fully proprietary US based service only company is to high given all circumstances. (Depending on how various things develop things like US export restricting OpenAI, even GPT-4, is a very real possibility companies can't ignore when doing long term product decisions.)
huytersd 3y ago
I’ll wait 5 seconds for the right code over 1 sec for bad code.
- sjwhevvvvvsj 3y ago
  Yes but if a 7b LLM will give you the same “Hello World” as the 70b, and that’s literally all you need, using a bigger model is just burning energy for no reason at all.
  - huytersd 3y ago
    The cost is fixed for me at least at this point so why would I choose the inferior version.
    - sjwhevvvvvsj 3y ago
      It’s not fixed whatsoever. Mistral 7B runs on a MacBook Air, and it’s free. Zero cost LLM, no network latency.

spaceman_2020 3y ago

Who cares about speed if you’re wrong? This isn’t a race to write the most lines of code or the most lines of text. It’s a race to write the most correct lines of code. I’ll wait half an hour for a response if I know I’m getting at least staff engineer level tier of code for every question

popinman322 3y ago
For the tasks my group is considering, even a 7B model is adequate. Sufficiently accurate responses can be fed into other systems downstream and cleaned up. Even code responses can benefit from this by restricting output tokens using the grammar of the target language, or iterating until the code compiles successfully. And for a decent number of LLM-enabled use cases the functionality unlocked by these models is novel. When you're going from 0 to 1 people will just be amazed that the product exists.
sjwhevvvvvsj 3y ago
Who says it’s wrong? I have very discrete tasks which involve resolving linguistic ambiguity and they can perform very well.
- mlnj 3y ago
  Exactly. Not everything is throwing large chunks of text to get complex questions answered. I love using the smaller models like Starling LM 7B and Mistral 7B have been enough for many tasks like you mentioned.
ein0p 3y ago
That’s the correct answer. Years ago I worked on inference efficiency on edge hardware at a startup. Time after time I saw that users vastly prefer slower, but more accurate and robust systems. Put succinctly: nobody cares how quick a model is if it doesn’t do a good job. Another thing I discovered is it can be very difficult to convince software engineers of this obvious fact.
- spacecadet 3y ago
  Having spent time on edge compute projects. This. Also, all the evidence is in this thread. Clearly people unhappy with wasting time on LLMs, when the time that was wasted was the result of obviously bad output.
  - sjwhevvvvvsj 3y ago
    People think LLM are all or nothing, like it’s either god-like AGI or it’s useless “hallucinating”. In reality you have to know the strengths and weaknesses of any tool, and small/fast LLM can do a tremendous amount within a fixed scope. The people at Mistral get this.
- Al-Khwarizmi 3y ago
  Less compute also means lower cost, though. I see how most people would prefer a better but slower model when price is equal, but I'm sure many prefer a worse $2/mo model over a better $20/mo model.
  - ein0p 3y ago
    That’s the thing I’m finding so hard to explain. Nobody would ever pay even $2 for a system that is worse at solving the problem. There is some baseline compute you need to deliver certain types of models. Going below that level for lower cost at the expense of accuracy and robustness is a fool’s errand. In LLMs it’s even worse. To make it concrete, for how I use LLMs I will not only not pay for anything with less capability than GPT4, I won’t even use it for free. It could be that other LLMs could perform well on narrow problems after fine tuning, but even then I’d prefer the model with the highest metrics, not the lowest inference cost.
    - sjwhevvvvvsj 3y ago
      So I think that’s a “your problem isn’t right for the tool” issue, not a “Mistral isn’t capable” issue.
dathinab 3y ago
Who care about getting better answers if you can't afford it, can't use it for legal reason or conclude that the risk associated with OpenAI now being a fully proprietary US based service only company is to high given all circumstances. (Depending on how various things develop things like US export restricting OpenAI, even GPT-4, is a very real possibility companies can't ignore when doing long term product decisions.)

huytersd 3y ago

I’ll wait 5 seconds for the right code over 1 sec for bad code.

sjwhevvvvvsj 3y ago
Yes but if a 7b LLM will give you the same “Hello World” as the 70b, and that’s literally all you need, using a bigger model is just burning energy for no reason at all.
- huytersd 3y ago
  The cost is fixed for me at least at this point so why would I choose the inferior version.
  - sjwhevvvvvsj 3y ago
    It’s not fixed whatsoever. Mistral 7B runs on a MacBook Air, and it’s free. Zero cost LLM, no network latency.