3 ms·

How about Mistral 7B? I saw this article recently: https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistral7B-on-Python-Code-With-A-Single-GPU---Vmlldzo1N

• redox99 3y ago
  Mistral 7B is very cool for its size. But unfortunately no open model is close to GPT4 as of right now.
  • tarruda 3y ago
    If the rumors around GPT4 being a mixture of expert models are true, the this comparison is not fair. What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
    • johnthewise 3y ago
      GPT4 being a mixture of experts is irrelevant imo like we don't care about how many layers there are in a network and how wide those layers are or which type of activation functions are actually used etc. all that matters are we can run it on a specific hardware and the results.
      • redox99 3y ago
        Exactly. I don't get why people (non AI researchers) discount MoE like they are cheating or fake parameters. Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.