6 ms·

This model has been in the top for quite a while, what's so good about it?

intellectronica 3y ago
Excellent performance for this model size and inference cost. Best model you can run on a device a small as a phone and get performance close to GPT-3.5 level. The structure and the training data are also interesting - sparse model using curated synthetic data to achieve much better accuracy than is achieved in models trained on random internet text.
- stavros 3y ago
  Is it really close to GPT-3.5 at 2.7B?
  - regularfry 3y ago
    It's 2.7B, not 1.1. In my experience it goes off the rails and starts generating nonsense after a few paragraphs, but I haven't dug too much into tweaking the kv cache params to see if that's controllable. It also needs a fair bit of prompt massaging to get it to do what you want. So no, not GPT3.5, but it's comfortably better than anything else in its size class.
    - eurekin 3y ago
      What are kv cache params?
      - regularfry 3y ago
        Key-value cache in the attention layers. There was a paper a little while back about how maintaining the first N tokens across an extended context helped an LLM keep sane for longer, and it turns out you can replicate it with the right CLI arguments to llama.cpp.
    - stavros 3y ago
      This sounds much more realistic, thanks!
    - realPtolemy 3y ago
      How is it compared to 7B LLaMA quantized to run on a raspberry pi?
      - regularfry 3y ago
        Probably similar token rates out of the box, although I havent done a straight comparison. Where they'll differ is in the sorts of questions they're good at. Llama2 was trained (broadly speaking) for knowledge, Phi-2 for reasoning. And bear in mind that you can quantise phi-2 down too. The starting point is f16.
      - jasonjmcghee 3y ago
        If you can run quantized 7B, nothing beats mistral and its fine tunes- like openhermes2.5
  - moffkalast 3y ago
    Close is a subjective term. Vicuna-33B from over half a year ago gets within 22 ELO of 3.5 in the arena leaderboard, but in practice the refusals are reducing 3.5 and other RLHFd models' ratings a lot and they're not even close. You can try Phi-2 with wasm here, I mostly just get gibberish out of it though: https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-demo?model=phi_1_5_q80 https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-... Mixtral is the only model that properly matches 3.5 at the moment.
    - stavros 3y ago
      That makes sense, thank you. There seems to be some inflation, as Mistral is supposed to be GPT-3.5-level, and Mixtral is supposed to be nearer GPT-4, but yeah, it sounds suspicious in practice, even though Mistral is very good.
      - moffkalast 3y ago
        Well in some things it totally can be to some extent, yes. You can almost certainly get a Mistral 7B fine tuned for a specific thing (e.g. coding) and it will likely be about as good as 3.5 in that specific thing (not a super high bar in objective terms). For all the other areas it may suffer in performance relative to its original self, but for some applications that's fine. As for GPT-4 it's about 120 ELO points [0] above Mixtral, and that's even the distilled turbo version. Not even close imo, especially when Mixtral is far less censored. Both 3.5 and 4 have changed drastically over the past year with continued fine tuning, quantization, etc. so what some people consider their level is not exactly a fixed point either. [0] The actual leaderboard I'm referencing, it has its biases but it's the most generally indicative thing available right now: https://chat.lmsys.org https://chat.lmsys.org
        regularfry 3y ago
        Mixtral is only 32 ELO ahead of the best 7B model on that leaderboard, although I suspect that might be understating the difference.
    - 3y ago
  - WiSaGaN 3y ago
    In my experience, it's great at its size, but obviously worse than mistral:7b-instruct-v0.2. Currently mixtral:8x7b-instruct-v0.1 is the lowest inference cost model at similar performance level of GPT-3.5.
- jasonjmcghee 3y ago
  Close to gpt-3.5? Because I’ve tried it and fine tuned variants and it’s been horrible. Next to useless on general tasks. No where near mistral 7 and absolutely not even close to gpt-3.5. Best 2.7b? sure. I have always thought of this model as something you fine tune for a very specific task or dataset. For others that don’t strongly disagree with the parent comment, can you point me to some examples?
OhNoNotAgain_99 3y ago
[dead]

Excellent performance for this model size and inference cost. Best model you can run on a device a small as a phone and get performance close to GPT-3.5 level. The structure and the training data are also interesting - sparse model using curated synthetic data to achieve much better accuracy than is achieved in models trained on random internet text.

stavros 3y ago
Is it really close to GPT-3.5 at 2.7B?
- regularfry 3y ago
  It's 2.7B, not 1.1. In my experience it goes off the rails and starts generating nonsense after a few paragraphs, but I haven't dug too much into tweaking the kv cache params to see if that's controllable. It also needs a fair bit of prompt massaging to get it to do what you want. So no, not GPT3.5, but it's comfortably better than anything else in its size class.
  - eurekin 3y ago
    What are kv cache params?
    - regularfry 3y ago
      Key-value cache in the attention layers. There was a paper a little while back about how maintaining the first N tokens across an extended context helped an LLM keep sane for longer, and it turns out you can replicate it with the right CLI arguments to llama.cpp.
  - stavros 3y ago
    This sounds much more realistic, thanks!
  - realPtolemy 3y ago
    How is it compared to 7B LLaMA quantized to run on a raspberry pi?
    - regularfry 3y ago
      Probably similar token rates out of the box, although I havent done a straight comparison. Where they'll differ is in the sorts of questions they're good at. Llama2 was trained (broadly speaking) for knowledge, Phi-2 for reasoning. And bear in mind that you can quantise phi-2 down too. The starting point is f16.
    - jasonjmcghee 3y ago
      If you can run quantized 7B, nothing beats mistral and its fine tunes- like openhermes2.5
- moffkalast 3y ago
  Close is a subjective term. Vicuna-33B from over half a year ago gets within 22 ELO of 3.5 in the arena leaderboard, but in practice the refusals are reducing 3.5 and other RLHFd models' ratings a lot and they're not even close. You can try Phi-2 with wasm here, I mostly just get gibberish out of it though: https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-demo?model=phi_1_5_q80 https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-... Mixtral is the only model that properly matches 3.5 at the moment.
  - stavros 3y ago
    That makes sense, thank you. There seems to be some inflation, as Mistral is supposed to be GPT-3.5-level, and Mixtral is supposed to be nearer GPT-4, but yeah, it sounds suspicious in practice, even though Mistral is very good.
    - moffkalast 3y ago
      Well in some things it totally can be to some extent, yes. You can almost certainly get a Mistral 7B fine tuned for a specific thing (e.g. coding) and it will likely be about as good as 3.5 in that specific thing (not a super high bar in objective terms). For all the other areas it may suffer in performance relative to its original self, but for some applications that's fine. As for GPT-4 it's about 120 ELO points [0] above Mixtral, and that's even the distilled turbo version. Not even close imo, especially when Mixtral is far less censored. Both 3.5 and 4 have changed drastically over the past year with continued fine tuning, quantization, etc. so what some people consider their level is not exactly a fixed point either. [0] The actual leaderboard I'm referencing, it has its biases but it's the most generally indicative thing available right now: https://chat.lmsys.org https://chat.lmsys.org
      - regularfry 3y ago
        Mixtral is only 32 ELO ahead of the best 7B model on that leaderboard, although I suspect that might be understating the difference.
  - 3y ago
- WiSaGaN 3y ago
  In my experience, it's great at its size, but obviously worse than mistral:7b-instruct-v0.2. Currently mixtral:8x7b-instruct-v0.1 is the lowest inference cost model at similar performance level of GPT-3.5.
jasonjmcghee 3y ago
Close to gpt-3.5? Because I’ve tried it and fine tuned variants and it’s been horrible. Next to useless on general tasks. No where near mistral 7 and absolutely not even close to gpt-3.5. Best 2.7b? sure. I have always thought of this model as something you fine tune for a very specific task or dataset. For others that don’t strongly disagree with the parent comment, can you point me to some examples?

OhNoNotAgain_99 3y ago

[dead]