19 ms·

Releasing 3B and 7B RedPajama

363 points by antimatter15 3y ago

ibitto 3y ago
I am really interested in knowing what people are using these smaller models for. I have seen a lot of projects on top of GPT-3.5 / GPT-4, but I have yet to see any using these smaller models.
wtarreau 3y ago
That's very interesting to perform basic tasks at reasonable speeds or to run on smaller systems. Unfortunately it's not of the many ones based on python and transformers, so all gained resources from the compact model are wasted by the heavy engine and ecosystem, and even a 4GB machine with 4G swap goes oom because the loaded data gets duplicated in memory using read() and malloc() :-( Let's wait for someone to port it to a cheaper and more powerful C-based engine like llama-cpp.
ftxbro 3y ago
With this one and mosaicml we now got so many of these consumer-gpu-sized models!
acapybara 3y ago
I've been following the RedPajama project closely and I must say, it's quite an impressive undertaking. The fact that it's all open-source, and the collaboration between various institutions, is nothing short of amazing. This shows the power of the open-source community in action, with a bunch of smart people coming together to build something truly remarkable. The 3B model, being super fast and accessible, is a game changer for a lot of us who may not have the latest hardware. I mean, running on an RTX 2070 that was released 5 years ago? That's pretty cool. As for the 7B model, it's great to see that it's already outperforming the Pythia 7B. The bigger dataset definitely seems to be making a difference here. I'm eager to see how far this project goes, and what kinda improvements we can expect in the coming weeks with the new RedPajama dataset they're working on. One thing I found interesting is the mention of differences between the LLaMA 7B and their replication. I'd love to learn more about those differences, as it could shed light on what's working well and what could be improved further.
- SeanAnderson 3y ago
  Sorry, excuse my ignorance, but why is having access to a 3B model a gamechanger? I played with a pirated 7B model a while back. My computer runs a 1080 TI - so it used to be good but now it's pretty old. The model ran with a reasonable number of tokens/sec, but the quality was just trash compared to what I'd grown used to with ChatGPT. It was a novelty I interacted with for just a single evening. I truly don't understand the use case for a 3B model with our current technologies. What are you going to use it for?
  - acapybara 3y ago
    Hey SeanAnderson, good question! While parameter count is certainly an important factor in model performance, it's not the only one. The RedPajama project is taking a more nuanced approach to understanding what makes a model perform well, and their focus on smaller models like the 3B is a big part of that. Sure, you may have played with a 7B model in the past, but that doesn't mean there's no use case for a smaller model like the 3B. In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. Plus, smaller models are generally faster and more accessible, which is always a plus.
    - hhh 3y ago
      is this comment generated by an LLM?
    - wokwokwok 3y ago
      > In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. So we are all in agreement here that a 3B model is fundamentally inferior to a larger model? Not that it doesn’t have uses; not that there’s no value in research in small models. Just, honestly, that these smaller models don’t have the capabilities of the larger models. It’d be good to be a direct acknowledgment of that, because it seems like you’re going out of your way to promote the “it’s fine to have a small model”; and it is, roughly speaking. Parameter count isn’t everything. Small models are accessible, you can easily fine tune them. They are interesting. …but, they are not as good, as far as I’m aware, in terms of output, in terms of general purpose function, as larger models.
andy_xor_andrew 3y ago
This is beyond exciting. Welcome to the new reality! On one hand, the resources required to run these models continues falling dramatically, thanks to the techniques discovered by researchers: GPTQ quantizing down to 4, 3, 2, even 1 bits! model pruning! hybrid vram offloading! better, more efficient architectures! 1-click finetuning on consumer hardware! Of course, the free lunches won't last forever, and this will level off, but it's still incredible. And on the other side of the coin, the power of all computing devices continues its ever-upward exponential growth. So you have a continuous lowering of requirements, combined with a continuous increase in available power... surely these two trends will collide, and I can only imagine what this stuff will be like at that intersection.
- visarga 3y ago
  At that intersection is the "Good Enough Model" that can solve 95% of our needs in full privacy and with complete customisability. The key point is being easy to run on every device. We'll still use proprietary, expensive models for the rest of 5%.
- acapybara 3y ago
  [flagged]
  - quickthrower2 3y ago
    This of course is making AI more Open, the final piece is for people to start packing these in Windows and Mac installers so that the average computer user can make use of them. And for them to run on the crappy graphics card you are likely to get with your default configuration of PC! Or for these to be at least available in every cloud and well understood so you are just paying AWS for compute but not secret sauce (like Kubernetes for example which lets you walk and there is genuine competition)
  - Mike_12345 3y ago
    [flagged]
    - 3y ago
rawrmaan 3y ago
There was a lot of detail and data in here, but it's not very useful to me because all of the comparisons are to things I have no experience with. There's really only one thing I care about: How does this compare to GPT-4? I have no use for models that aren't at that level. Even though this almost definitely isn't at that level, it's hard to know how close or far it is from the data presented.
- Joeri 3y ago
  None of the 3B and 7B models are at ChatGPT’s level, let alone GPT-4. The 13B models start doing really interesting things, but you don’t get near ChatGPT results until you move up to the best 30B and 65B models, which require beefier hardware. Nothing out there right now approximates GPT-4. The big story here for me is that the difference in training set is what makes the difference in quality. There is no secret sauce, the open source architectures do well, provided you give them a large and diverse enough training set. That would mean it is just a matter of pooling resources to train really capable open source models. That makes what RedPajama is doing, compiling the best open dataset, very important for the future of high quality open source LLM’s. If you want to play around with this yourself you can install oobabooga and figure out what model fits your hardware from the locallama reddit wiki. The llama.cpp 7B and 13B models can be run on CPU if you have enough RAM. I’ve had lots of fun talking to 7B and 13B alpaca and vicuna models running locally. https://www.reddit.com/r/LocalLLaMA/wiki/models/ https://www.reddit.com/r/LocalLLaMA/wiki/models/
  - Semaphor 3y ago
    > The llama.cpp 7B and 13B models can be run on CPU if you have enough RAM. Bigger ones as well, you just have to wait longer. Nothing for real time usage, but if you can wait 10-20 minutes, you can use them on CPU.
    - int_19h 3y ago
      It's not even that bad. Core i7-12700K with DDR5 gives me ~1 word per second on llama-30b - that is fast enough for real-time chat, with some patience. And things are even better on M1/M2 Macs.
knaik94 3y ago
I have been really impressed with the uncensored WizardLM I was playing with. Having a truely open uncensored model to work with is a really important research tool. Censorship of the training data and results in such a heavy handed way is not really possible without lowering the quality of all output. As the resouces required to train and fine tune these models becomes consumer handware friendly, I think we'll see a shift towards a bunch of smaller models. Open models like these also mean the results of securty and capability research is publicly available. Models like this one and the Replit code model will become the new base all open source models are based on. I am really looking forward to the gptj 4bit, cuda optimized 7b models, the others I have tested run fast on 2070max q and 16gb ram, I was getting ~7tokens/second. Lora can work directly with 4bit quantized models. While ggml, cpu models are very strong, I don't believe we're move away from gpu accelarated training and fine tuning anytime soon.
- regularfry 3y ago
  The thing is that anything that benefits the bottom end also should reflect up and help the top end too, if they're paying attention.
ftxbro 3y ago
So I tried RedPajama-INCITE-Instruct-7B-v0.1 and the AutoModelForCausalLM.from_pretrained(...) call takes two minutes every time. My GPU is big enough. I don't know why it's so slow. I feel like it's somehow precomputing stuff that can be used across queries, and I had hoped that this stuff would have already been precomputed on the disk and I could just load it up.
sphars 3y ago
Slightly off-topic, but as the parent of a toddler, I got a bit of a chuckle out of the name. It's based off the children's book series of "Llama Llama Red Pajama"
- petesergeant 3y ago
  It had put me in mind of the Ogden Nash poem: The one-l lama, He's a priest. The two-l llama, He's a beast. And I will bet A silk pajama There isn't any Three-l lllama.
  - dllthomas 3y ago
    "*The author's attention has been called to a type of conflagration known as a three-alarmer. Pooh."
- blurbleblurble 3y ago
  Not off topic at all
- innagadadavida 3y ago
  Founder ex-apple Siri search. Had a baby a couple of years ago. Not too surprising to me :)
- elkos 3y ago
  Thanks. As non-native English speaker (while though a parent of a toddler too) I wasn't familiar with the book series.
  - Auracle 3y ago
    As the father of an 18 month old daughter that likes the book, I have it memorized.
- dllthomas 3y ago
  I'm holding out for the MadAtMama model.
mirker 3y ago
Does anyone have experience using these open source models in production?
- flatiron 3y ago
  Doubtful since they were released yesterday. That being said I will be deploying something to our lab this week to play with.
practice9 3y ago
Models replicating LLaMA are cool, but they are all missing proper multilingual support, which GPT-3.5 is quite good at.
- tyfon 3y ago
  Llama 65B is actually quite decent in other languages. I can just barely fit it in memory though with my 128 gb ram. Usually I run the 8 bit quantized version that use 80, but even the 4 and 3 but are ok compared to the fp16 30B version.
- mirekrusin 3y ago
  IMHO multilingual support would just pollute precious available estate in those models. Why not use it in english and use another one for translation?
  - espadrine 3y ago
    It depends on your use. LLaMA’s main issue is that its license prevents commercial use. If you want to use a LLM inside of a product, you may need to internationalize it at some point, so multilingual support matters.
  - viraptor 3y ago
    That would work if all information is available in English as the primary language. That's not the case though. You may be missing out on interesting information if you're skipping other languages.
born-jre 3y ago
i also wonder how powerful will 3b model will be ? can it act as a prompt router where it can make API call to ChatGPT or other specified model for actual processing. its probably possible to do this with langchain but i have not tried it yet.
nico 3y ago
idea: linked parameters / models tree build a model that can change the number of parameters in the vicinity of some meaning, effectively increasing the local resolution around that meaning so parameter space becomes linked-parameter space, between models links could be pruned based on activation frequency another way of seeing the concept is a tree of models/llms and one additional model/llm that all it does is manage the tree (ie. build it as it goes, use it to infer, prune it, etc) Or is it too dumb what I’m saying?

363 points by antimatter15 3y ago

I am really interested in knowing what people are using these smaller models for. I have seen a lot of projects on top of GPT-3.5 / GPT-4, but I have yet to see any using these smaller models.

wtarreau 3y ago

That's very interesting to perform basic tasks at reasonable speeds or to run on smaller systems. Unfortunately it's not of the many ones based on python and transformers, so all gained resources from the compact model are wasted by the heavy engine and ecosystem, and even a 4GB machine with 4G swap goes oom because the loaded data gets duplicated in memory using read() and malloc() :-( Let's wait for someone to port it to a cheaper and more powerful C-based engine like llama-cpp.

ftxbro 3y ago

With this one and mosaicml we now got so many of these consumer-gpu-sized models!

acapybara 3y ago

I've been following the RedPajama project closely and I must say, it's quite an impressive undertaking. The fact that it's all open-source, and the collaboration between various institutions, is nothing short of amazing. This shows the power of the open-source community in action, with a bunch of smart people coming together to build something truly remarkable. The 3B model, being super fast and accessible, is a game changer for a lot of us who may not have the latest hardware. I mean, running on an RTX 2070 that was released 5 years ago? That's pretty cool. As for the 7B model, it's great to see that it's already outperforming the Pythia 7B. The bigger dataset definitely seems to be making a difference here. I'm eager to see how far this project goes, and what kinda improvements we can expect in the coming weeks with the new RedPajama dataset they're working on. One thing I found interesting is the mention of differences between the LLaMA 7B and their replication. I'd love to learn more about those differences, as it could shed light on what's working well and what could be improved further.

SeanAnderson 3y ago
Sorry, excuse my ignorance, but why is having access to a 3B model a gamechanger? I played with a pirated 7B model a while back. My computer runs a 1080 TI - so it used to be good but now it's pretty old. The model ran with a reasonable number of tokens/sec, but the quality was just trash compared to what I'd grown used to with ChatGPT. It was a novelty I interacted with for just a single evening. I truly don't understand the use case for a 3B model with our current technologies. What are you going to use it for?
- acapybara 3y ago
  Hey SeanAnderson, good question! While parameter count is certainly an important factor in model performance, it's not the only one. The RedPajama project is taking a more nuanced approach to understanding what makes a model perform well, and their focus on smaller models like the 3B is a big part of that. Sure, you may have played with a 7B model in the past, but that doesn't mean there's no use case for a smaller model like the 3B. In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. Plus, smaller models are generally faster and more accessible, which is always a plus.
  - hhh 3y ago
    is this comment generated by an LLM?
  - wokwokwok 3y ago
    > In fact, having a performant, smaller model is a game changer for a lot of applications that don't require the massive scale of the larger models. So we are all in agreement here that a 3B model is fundamentally inferior to a larger model? Not that it doesn’t have uses; not that there’s no value in research in small models. Just, honestly, that these smaller models don’t have the capabilities of the larger models. It’d be good to be a direct acknowledgment of that, because it seems like you’re going out of your way to promote the “it’s fine to have a small model”; and it is, roughly speaking. Parameter count isn’t everything. Small models are accessible, you can easily fine tune them. They are interesting. …but, they are not as good, as far as I’m aware, in terms of output, in terms of general purpose function, as larger models.

andy_xor_andrew 3y ago

This is beyond exciting. Welcome to the new reality! On one hand, the resources required to run these models continues falling dramatically, thanks to the techniques discovered by researchers: GPTQ quantizing down to 4, 3, 2, even 1 bits! model pruning! hybrid vram offloading! better, more efficient architectures! 1-click finetuning on consumer hardware! Of course, the free lunches won't last forever, and this will level off, but it's still incredible. And on the other side of the coin, the power of all computing devices continues its ever-upward exponential growth. So you have a continuous lowering of requirements, combined with a continuous increase in available power... surely these two trends will collide, and I can only imagine what this stuff will be like at that intersection.

visarga 3y ago
At that intersection is the "Good Enough Model" that can solve 95% of our needs in full privacy and with complete customisability. The key point is being easy to run on every device. We'll still use proprietary, expensive models for the rest of 5%.
acapybara 3y ago
[flagged]
- quickthrower2 3y ago
  This of course is making AI more Open, the final piece is for people to start packing these in Windows and Mac installers so that the average computer user can make use of them. And for them to run on the crappy graphics card you are likely to get with your default configuration of PC! Or for these to be at least available in every cloud and well understood so you are just paying AWS for compute but not secret sauce (like Kubernetes for example which lets you walk and there is genuine competition)
- Mike_12345 3y ago
  [flagged]
  - 3y ago

rawrmaan 3y ago

There was a lot of detail and data in here, but it's not very useful to me because all of the comparisons are to things I have no experience with. There's really only one thing I care about: How does this compare to GPT-4? I have no use for models that aren't at that level. Even though this almost definitely isn't at that level, it's hard to know how close or far it is from the data presented.

Joeri 3y ago
None of the 3B and 7B models are at ChatGPT’s level, let alone GPT-4. The 13B models start doing really interesting things, but you don’t get near ChatGPT results until you move up to the best 30B and 65B models, which require beefier hardware. Nothing out there right now approximates GPT-4. The big story here for me is that the difference in training set is what makes the difference in quality. There is no secret sauce, the open source architectures do well, provided you give them a large and diverse enough training set. That would mean it is just a matter of pooling resources to train really capable open source models. That makes what RedPajama is doing, compiling the best open dataset, very important for the future of high quality open source LLM’s. If you want to play around with this yourself you can install oobabooga and figure out what model fits your hardware from the locallama reddit wiki. The llama.cpp 7B and 13B models can be run on CPU if you have enough RAM. I’ve had lots of fun talking to 7B and 13B alpaca and vicuna models running locally. https://www.reddit.com/r/LocalLLaMA/wiki/models/ https://www.reddit.com/r/LocalLLaMA/wiki/models/
- Semaphor 3y ago
  > The llama.cpp 7B and 13B models can be run on CPU if you have enough RAM. Bigger ones as well, you just have to wait longer. Nothing for real time usage, but if you can wait 10-20 minutes, you can use them on CPU.
  - int_19h 3y ago
    It's not even that bad. Core i7-12700K with DDR5 gives me ~1 word per second on llama-30b - that is fast enough for real-time chat, with some patience. And things are even better on M1/M2 Macs.

knaik94 3y ago

I have been really impressed with the uncensored WizardLM I was playing with. Having a truely open uncensored model to work with is a really important research tool. Censorship of the training data and results in such a heavy handed way is not really possible without lowering the quality of all output. As the resouces required to train and fine tune these models becomes consumer handware friendly, I think we'll see a shift towards a bunch of smaller models. Open models like these also mean the results of securty and capability research is publicly available. Models like this one and the Replit code model will become the new base all open source models are based on. I am really looking forward to the gptj 4bit, cuda optimized 7b models, the others I have tested run fast on 2070max q and 16gb ram, I was getting ~7tokens/second. Lora can work directly with 4bit quantized models. While ggml, cpu models are very strong, I don't believe we're move away from gpu accelarated training and fine tuning anytime soon.

regularfry 3y ago
The thing is that anything that benefits the bottom end also should reflect up and help the top end too, if they're paying attention.

ftxbro 3y ago

So I tried RedPajama-INCITE-Instruct-7B-v0.1 and the AutoModelForCausalLM.from_pretrained(...) call takes two minutes every time. My GPU is big enough. I don't know why it's so slow. I feel like it's somehow precomputing stuff that can be used across queries, and I had hoped that this stuff would have already been precomputed on the disk and I could just load it up.

sphars 3y ago

Slightly off-topic, but as the parent of a toddler, I got a bit of a chuckle out of the name. It's based off the children's book series of "Llama Llama Red Pajama"

petesergeant 3y ago
It had put me in mind of the Ogden Nash poem: The one-l lama, He's a priest. The two-l llama, He's a beast. And I will bet A silk pajama There isn't any Three-l lllama.
- dllthomas 3y ago
  "*The author's attention has been called to a type of conflagration known as a three-alarmer. Pooh."
blurbleblurble 3y ago
Not off topic at all
innagadadavida 3y ago
Founder ex-apple Siri search. Had a baby a couple of years ago. Not too surprising to me :)
elkos 3y ago
Thanks. As non-native English speaker (while though a parent of a toddler too) I wasn't familiar with the book series.
- Auracle 3y ago
  As the father of an 18 month old daughter that likes the book, I have it memorized.
dllthomas 3y ago
I'm holding out for the MadAtMama model.

mirker 3y ago

Does anyone have experience using these open source models in production?

flatiron 3y ago
Doubtful since they were released yesterday. That being said I will be deploying something to our lab this week to play with.

practice9 3y ago

Models replicating LLaMA are cool, but they are all missing proper multilingual support, which GPT-3.5 is quite good at.

tyfon 3y ago
Llama 65B is actually quite decent in other languages. I can just barely fit it in memory though with my 128 gb ram. Usually I run the 8 bit quantized version that use 80, but even the 4 and 3 but are ok compared to the fp16 30B version.
mirekrusin 3y ago
IMHO multilingual support would just pollute precious available estate in those models. Why not use it in english and use another one for translation?
- espadrine 3y ago
  It depends on your use. LLaMA’s main issue is that its license prevents commercial use. If you want to use a LLM inside of a product, you may need to internationalize it at some point, so multilingual support matters.
- viraptor 3y ago
  That would work if all information is available in English as the primary language. That's not the case though. You may be missing out on interesting information if you're skipping other languages.

born-jre 3y ago

i also wonder how powerful will 3b model will be ? can it act as a prompt router where it can make API call to ChatGPT or other specified model for actual processing. its probably possible to do this with langchain but i have not tried it yet.

nico 3y ago

idea: linked parameters / models tree build a model that can change the number of parameters in the vicinity of some meaning, effectively increasing the local resolution around that meaning so parameter space becomes linked-parameter space, between models links could be pruned based on activation frequency another way of seeing the concept is a tree of models/llms and one additional model/llm that all it does is manage the tree (ie. build it as it goes, use it to infer, prune it, etc) Or is it too dumb what I’m saying?