5 ms·

Is there a reason projects seem to be standardizing on specific parameter sizes within larger buckets? E.g. I only ever see news about 7B models, not 6B or 8B.

by jsnell 3y ago

Is there a reason projects seem to be standardizing on specific parameter sizes within larger buckets? E.g. I only ever see news about 7B models, not 6B or 8B. Are these sizes somehow optimal? Is it about getting as close to resource (memory?) breakpoints as possible without exceeding them? Is it to make comparisons between models simpler by removing one variable?

riskable 3y ago
Not an expert but I'm pretty sure it has to do with how much VRAM you need in your GPU in order to process them efficiently. Last time I was reading about the sizes someone mentioned that 8B was just barely too big for their GPU.
- aidenn0 3y ago
  Seems like 6B would still be useful if I want to run it on my GPU without exiting firefox.
  - londons_explore 3y ago
    Kinda lame that applications can't be told "yo, your gpu buffer has now been moved back to RAM".
    - selcuka 3y ago
      With Apple Silicon there is no separation between GPU RAM and CPU RAM, so the whole model can be loaded into RAM and processed by GPU (Metal).
- brucethemoose2 3y ago
  This is not really true anymore, as the "consumer GPU" backends have very flexible quantization. Llama.cpp has like a dozen steps between 2 and 6 bit, and exLlamav2 will literally do an arbitrary decimal bpw. It sort of matters with bigger models trying to squeeze into a server GPU, with the (currently) inflexible vLLM 4-bit quantization. I think its just a standard set by Llama.
brucethemoose2 3y ago
There are some mad lads making different sizes of llama by "grafting" attention heads from one model onto another and finetuning a bit to stablize the transplant. For instance: https://huggingface.co/models?sort=modified&search=20B https://huggingface.co/models?sort=modified&search=20B Its very experimental, but apparently the 20B models are actually improving on 13B.
- semi-extrinsic 3y ago
  Just the language being used here is amazing.
  - brucethemoose2 3y ago
    From the original effort linked above: > This is Llama 2 13b with some additional attention heads from original-flavor Llama 33b frankensteined on. And the lingo gets weirder, with an experimental merging method, for instsnce, being called "SLERP"
    - skykooler 3y ago
      Is it something other than the standard meaning of "slerp" as spherical linear interpolation? That's not exactly new, it's been around since the mid-1980s.
- londons_explore 3y ago
  Any place people doing such grafting are congregating? I've often pondered if taking some random chunk of weights from the middle of a trained model, and dumping it into some totally different model might perform better than random initialization when the scale gets big enough.
  - brucethemoose2 3y ago
    I dunno. Probably the Kobold, Pygmalion, or AI Collective Discord, if I were to guess. The first effort I am aware of is here: https://huggingface.co/chargoddard/llama2-22b https://huggingface.co/chargoddard/llama2-22b Being in the Discord age, lots of the discussion about cool llm stuff is fragmented and buried. I am in these discords, and I only know because I ran into it on HuggingFace.
godelski 3y ago
Short: The short answer is that it is hard to compare models so to make it easier we compare parameters. Part of the answer of why we do it is because it also helps show scaling. (As far as I'm aware) The parameters __are not__ optimal, and we have no idea what actually that would mean. Longer: The longer answer is that comparing models is really fucking hard and how we tend to do it in the real world is not that great. You have to think of papers and experiments as proxies, but proxies for what? There's so many things that you need to compare a model on and it is actually really difficult to convey. Are you just trying to get the best performance? Are you trying to demonstrate a better architecture? Are you increasing speed? Are you increasing generalization (note the difference from performance)? And so on. Then we need to get into the actual metrics. What do the metrics mean? What are their limitations? What do they actually convey? These parts are unfortunately not asked as much but note that all metrics are models too (everything you touch is "a model"), and remember that "all models are wrong." It's important to remember that there are hundreds or thousands of metrics out there and they all have different biases and limitations, with no single metric being able to properly convey how good a model is at any task you choose. There is no "best language model" metric, nor are there even more specific "best at writing leet code style problems in python" metrics (though we'd be better at capturing that than the former question). Metrics are only guides and you must be truly aware of their limitations to properly evaluate (especially when we talk about high dimensions). This is why I rant about math in ML: You don't need to know math to make a good model, but you do need to know math to know why a model is wrong. Parameters (along with GMACs, which is dominating the FLOPs camp. Similarly inference speeds have become common place) only started to be included as common practice in the last few years and still not in every subject (tends to be around the transformer projects, both language and vision). As a quick example of why we want them, check out DDP vs iDDPM. You wouldn't know that the models are about 60% different in parameter size when comparing (Table 3). In fact, you're going to have a hard time noticing the difference unless you read both very carefully as they're both one liners (or just load the models. fucking tensorflow 1.15...). Does it seem fair to compare these two models? Obviously it depends, right? Is it fair to compare LLaMA 2 70B to LLaMA 2 7B? It both is and isn't. It entirely depends on what your needs are, but these are quite difficult to accurately capture. If my needs are to run on device in a mobile phone 7B probably wins hands down, but this would flip if I'm running on a server. The thing is that we just need to be clear about our goals, right? The more specific we can get about goals, the more specific we can get around comparing. But there's also weird effects that the metrics (remember, these are models too. Ask models of what) we use aren't entirely capturing. You may notice that some models have "better scores" but don't seem to work as well in real world use, right? Those are limitations of the metrics. While a better negative log likelihood/entropy score correlates well with being a high performing language model, it does not __mean__ a high performing language model. Entropy is a capture of information (but make sure not to conflate with the vernacular definition). These models are also very specifically difficult to evaluate given that they are trained and tested on different datasets (I absolutely rage here because non-hacking can't be verified) as well as the alignment done post process. This all gets incredibly complex and the honest truth here is that I don't think there is enough discussion around the topic of what a clusterfuck it is to compare models. Hell, it is hard to even compare more simple models doing more simple tasks like even just classifying MNIST numbers. Much harder than you might think. And don't get me started on out of distribution, generalization, and/or alignment. I would just say: if you're a layman, just watch and judge by how useful the tools are to you as a user -- be excited about the progress but don't let people sell you snake oil; if you're a researcher, why the fuck are we getting more lazy in evaluating works as the complexity of evaluation is exponentially increasing -- seriously, what the fuck is wrong with us?
jstarfish 3y ago
7B seems to be the limit of what people can comfortably fit in last-gen GPUs, having ~6GB of VRAM. It's also the lower acceptable boundary of coherence for generative text. There are some major releases at lesser parameter counts though. Databricks' Dolly had a 3B model, and Microsoft's Orca also had a recent 3B release. They're both abysmal at generating text, but I find them quick and useful for reductive tasks ("summarize this," "extract keywords from," etc.). (I like to treat parameter count as a measure of age/WIS/INT. For this question, do I need the wisdom of a 7-year old, a 13-year old, a 30-year old, etc. 3B is like polling preschoolers at daycare.)
dragonwriter 3y ago
> Is there a reason projects seem to be standardizing on specific parameter sizes within larger buckets? AFAICT, it is because science: most of them are research artifacts and intended to support further research, and the fewer of parameter count, model architecture, training set, etc., that change substantially between models, the easier it is to evaluate the effects each element changing.

by jsnell 3y ago

riskable 3y ago

Not an expert but I'm pretty sure it has to do with how much VRAM you need in your GPU in order to process them efficiently. Last time I was reading about the sizes someone mentioned that 8B was just barely too big for their GPU.

aidenn0 3y ago
Seems like 6B would still be useful if I want to run it on my GPU without exiting firefox.
- londons_explore 3y ago
  Kinda lame that applications can't be told "yo, your gpu buffer has now been moved back to RAM".
  - selcuka 3y ago
    With Apple Silicon there is no separation between GPU RAM and CPU RAM, so the whole model can be loaded into RAM and processed by GPU (Metal).
brucethemoose2 3y ago
This is not really true anymore, as the "consumer GPU" backends have very flexible quantization. Llama.cpp has like a dozen steps between 2 and 6 bit, and exLlamav2 will literally do an arbitrary decimal bpw. It sort of matters with bigger models trying to squeeze into a server GPU, with the (currently) inflexible vLLM 4-bit quantization. I think its just a standard set by Llama.

brucethemoose2 3y ago

There are some mad lads making different sizes of llama by "grafting" attention heads from one model onto another and finetuning a bit to stablize the transplant. For instance: https://huggingface.co/models?sort=modified&search=20B https://huggingface.co/models?sort=modified&search=20B Its very experimental, but apparently the 20B models are actually improving on 13B.

semi-extrinsic 3y ago
Just the language being used here is amazing.
- brucethemoose2 3y ago
  From the original effort linked above: > This is Llama 2 13b with some additional attention heads from original-flavor Llama 33b frankensteined on. And the lingo gets weirder, with an experimental merging method, for instsnce, being called "SLERP"
  - skykooler 3y ago
    Is it something other than the standard meaning of "slerp" as spherical linear interpolation? That's not exactly new, it's been around since the mid-1980s.
londons_explore 3y ago
Any place people doing such grafting are congregating? I've often pondered if taking some random chunk of weights from the middle of a trained model, and dumping it into some totally different model might perform better than random initialization when the scale gets big enough.
- brucethemoose2 3y ago
  I dunno. Probably the Kobold, Pygmalion, or AI Collective Discord, if I were to guess. The first effort I am aware of is here: https://huggingface.co/chargoddard/llama2-22b https://huggingface.co/chargoddard/llama2-22b Being in the Discord age, lots of the discussion about cool llm stuff is fragmented and buried. I am in these discords, and I only know because I ran into it on HuggingFace.

godelski 3y ago

Short: The short answer is that it is hard to compare models so to make it easier we compare parameters. Part of the answer of why we do it is because it also helps show scaling. (As far as I'm aware) The parameters __are not__ optimal, and we have no idea what actually that would mean. Longer: The longer answer is that comparing models is really fucking hard and how we tend to do it in the real world is not that great. You have to think of papers and experiments as proxies, but proxies for what? There's so many things that you need to compare a model on and it is actually really difficult to convey. Are you just trying to get the best performance? Are you trying to demonstrate a better architecture? Are you increasing speed? Are you increasing generalization (note the difference from performance)? And so on. Then we need to get into the actual metrics. What do the metrics mean? What are their limitations? What do they actually convey? These parts are unfortunately not asked as much but note that all metrics are models too (everything you touch is "a model"), and remember that "all models are wrong." It's important to remember that there are hundreds or thousands of metrics out there and they all have different biases and limitations, with no single metric being able to properly convey how good a model is at any task you choose. There is no "best language model" metric, nor are there even more specific "best at writing leet code style problems in python" metrics (though we'd be better at capturing that than the former question). Metrics are only guides and you must be truly aware of their limitations to properly evaluate (especially when we talk about high dimensions). This is why I rant about math in ML: You don't need to know math to make a good model, but you do need to know math to know why a model is wrong. Parameters (along with GMACs, which is dominating the FLOPs camp. Similarly inference speeds have become common place) only started to be included as common practice in the last few years and still not in every subject (tends to be around the transformer projects, both language and vision). As a quick example of why we want them, check out DDP vs iDDPM. You wouldn't know that the models are about 60% different in parameter size when comparing (Table 3). In fact, you're going to have a hard time noticing the difference unless you read both very carefully as they're both one liners (or just load the models. fucking tensorflow 1.15...). Does it seem fair to compare these two models? Obviously it depends, right? Is it fair to compare LLaMA 2 70B to LLaMA 2 7B? It both is and isn't. It entirely depends on what your needs are, but these are quite difficult to accurately capture. If my needs are to run on device in a mobile phone 7B probably wins hands down, but this would flip if I'm running on a server. The thing is that we just need to be clear about our goals, right? The more specific we can get about goals, the more specific we can get around comparing. But there's also weird effects that the metrics (remember, these are models too. Ask models of what) we use aren't entirely capturing. You may notice that some models have "better scores" but don't seem to work as well in real world use, right? Those are limitations of the metrics. While a better negative log likelihood/entropy score correlates well with being a high performing language model, it does not __mean__ a high performing language model. Entropy is a capture of information (but make sure not to conflate with the vernacular definition). These models are also very specifically difficult to evaluate given that they are trained and tested on different datasets (I absolutely rage here because non-hacking can't be verified) as well as the alignment done post process. This all gets incredibly complex and the honest truth here is that I don't think there is enough discussion around the topic of what a clusterfuck it is to compare models. Hell, it is hard to even compare more simple models doing more simple tasks like even just classifying MNIST numbers. Much harder than you might think. And don't get me started on out of distribution, generalization, and/or alignment. I would just say: if you're a layman, just watch and judge by how useful the tools are to you as a user -- be excited about the progress but don't let people sell you snake oil; if you're a researcher, why the fuck are we getting more lazy in evaluating works as the complexity of evaluation is exponentially increasing -- seriously, what the fuck is wrong with us?

jstarfish 3y ago

7B seems to be the limit of what people can comfortably fit in last-gen GPUs, having ~6GB of VRAM. It's also the lower acceptable boundary of coherence for generative text. There are some major releases at lesser parameter counts though. Databricks' Dolly had a 3B model, and Microsoft's Orca also had a recent 3B release. They're both abysmal at generating text, but I find them quick and useful for reductive tasks ("summarize this," "extract keywords from," etc.). (I like to treat parameter count as a measure of age/WIS/INT. For this question, do I need the wisdom of a 7-year old, a 13-year old, a 30-year old, etc. 3B is like polling preschoolers at daycare.)

dragonwriter 3y ago

> Is there a reason projects seem to be standardizing on specific parameter sizes within larger buckets? AFAICT, it is because science: most of them are research artifacts and intended to support further research, and the fewer of parameter count, model architecture, training set, etc., that change substantially between models, the easier it is to evaluate the effects each element changing.