4 ms·

Hey, asking any experts here, what are their first thoughts in the significance of this? IE, is this comparable to any other model released, or are there signi

by stale2002 3y ago

Hey, asking any experts here, what are their first thoughts in the significance of this? IE, is this comparable to any other model released, or are there significant metric differences that make it better for certain usecases? The only thing I see, of the top of my head, is that it is a very large model, and I don't think any models of similar size have been released.

whimsicalism 3y ago
seems like a large undertrained model, not that exciting imo compared to mixtral it is also not the biggest model oss, switch transformer was released years ago and is larger and similarly undertrained
brucethemoose2 3y ago
Tests are not out yet, but: - It's very large, yes. - It's a base model, so its not really practical to use without further finetuning. - Based on Grok-1 API performance (which itself is probably a finetune) its... not great at all.
Me1000 3y ago
Not an expert by any means, but I like learning about this stuff and I play with a lot of open weight models. I’d say the significance is that it happened. It’s by far the largest open weight model I’ve seen. But I’m not sure why you’d use it over a model like Mixtral, which seems to perform about the same at like 1/6th the size. But I welcome any contribution to the open weight LLM community. Hopefully people will learn something interesting with this model. And I hope they keep releasing new versions!
- MichaelRazum 3y ago
  If I may ask, how do you load such big models? 300gb seems like a lot to play around with.
  - Me1000 3y ago
    You're right, this model is going to be too big for most people to play around with. But to answer your question I have a 128GB of RAM in my M3 MacBook Pro, so I can use most of that for GPU inferencing. But still, this model is going to need to be heavily quantized for me to be able to use it. (fwiw, I probably wont try this one) In the next week or two I expect we'll see a GGUF version of the weights (might need to wait for a patch to llama.cpp first), and someone will release super small quantizations of it. I suspect my computer might be able to run a 3 bit quant, but it might need to go down to 2 bits to have any kind of reasonable context length. But with quants that small I'd expect the model's performance to degrade well below that of Mixtral, so it probably isn't really even worth using. But we'll see; quantization is weird, some models perform better than others when quantized.
    - TMWNN 3y ago
      >In the next week or two I expect we'll see a GGUF version of the weights (might need to wait for a patch to llama.cpp first), and someone will release super small quantizations of it. How quickly are new models available through Ollama?
      - cjbprime 3y ago
        Few days max.
      - Me1000 3y ago
        Ollama is just a wrapper around llama.cpp, so when the gguf model files come out it'll be able to run on Ollama (assuming no llama.cpp patch is needed, but even if it is ollama is usually good at getting those updates out pretty quickly).
    - zozbot234 3y ago
      A top-of-the-line Mac Studio Ultra maxes out at 192GB currently. This is also a MoE model, so only a fraction of parameters have to be in RAM.

whimsicalism 3y ago

seems like a large undertrained model, not that exciting imo compared to mixtral it is also not the biggest model oss, switch transformer was released years ago and is larger and similarly undertrained

brucethemoose2 3y ago

Tests are not out yet, but: - It's very large, yes. - It's a base model, so its not really practical to use without further finetuning. - Based on Grok-1 API performance (which itself is probably a finetune) its... not great at all.

Me1000 3y ago

Not an expert by any means, but I like learning about this stuff and I play with a lot of open weight models. I’d say the significance is that it happened. It’s by far the largest open weight model I’ve seen. But I’m not sure why you’d use it over a model like Mixtral, which seems to perform about the same at like 1/6th the size. But I welcome any contribution to the open weight LLM community. Hopefully people will learn something interesting with this model. And I hope they keep releasing new versions!

MichaelRazum 3y ago
If I may ask, how do you load such big models? 300gb seems like a lot to play around with.
- Me1000 3y ago
  You're right, this model is going to be too big for most people to play around with. But to answer your question I have a 128GB of RAM in my M3 MacBook Pro, so I can use most of that for GPU inferencing. But still, this model is going to need to be heavily quantized for me to be able to use it. (fwiw, I probably wont try this one) In the next week or two I expect we'll see a GGUF version of the weights (might need to wait for a patch to llama.cpp first), and someone will release super small quantizations of it. I suspect my computer might be able to run a 3 bit quant, but it might need to go down to 2 bits to have any kind of reasonable context length. But with quants that small I'd expect the model's performance to degrade well below that of Mixtral, so it probably isn't really even worth using. But we'll see; quantization is weird, some models perform better than others when quantized.
  - TMWNN 3y ago
    >In the next week or two I expect we'll see a GGUF version of the weights (might need to wait for a patch to llama.cpp first), and someone will release super small quantizations of it. How quickly are new models available through Ollama?
    - cjbprime 3y ago
      Few days max.
    - Me1000 3y ago
      Ollama is just a wrapper around llama.cpp, so when the gguf model files come out it'll be able to run on Ollama (assuming no llama.cpp patch is needed, but even if it is ollama is usually good at getting those updates out pretty quickly).
  - zozbot234 3y ago
    A top-of-the-line Mac Studio Ultra maxes out at 192GB currently. This is also a MoE model, so only a fraction of parameters have to be in RAM.