6 ms·

Pretty exciting to see the democratization of this tech. The next wave of innovation seems like it’ll be much less centralized into a handful of big companies.

Pretty exciting to see the democratization of this tech. The next wave of innovation seems like it’ll be much less centralized into a handful of big companies. Even more powerful closed door LLMs will appear, but already so much utility out of these that can run on consumer hardware.

yieldcrv 4y ago
Feels like we’ll get to “good enough” in a matter of weeks. Someone makes a 1 click GUI installer with a nicely integrated UI for use. No command line use at all. We’ll get model and weight distributors for updated knowledge.
bogwog 4y ago
Only if one of those handful of big companies decides to release their models for commercial use. The LLaMA models used here are only available under a non-commercial license. Can’t build anything significant with leaked models. Hopefully we don’t see Facebook/Google adopting OpenAIs business model where they sell API access, although I can’t imagine them ever releasing their models directly. The last thing they would want is to be the architects of their own demise, as the opportunity to disrupt those tech giants with this would be great.
- adam_arthur 4y ago
  The techniques to train and reproduce similarly performing models seem fairly widely understood and replicable at this point. I do agree that large orgs are likely to have the best models, but a lot of the low hanging fruit/initial utility is already covered by models that run on consumer hardware
  - bitL 4y ago
    Can you post a link on how to train a gigantic LLM on consumer hardware with all detailed steps please? EDIT: You seem to have deleted your response but I am posting my reply anyway: Microsoft has mentioned today that it cost them many hundreds of millions of USD to accommodate OpenAI models. I studied NLP at Stanford including LLMs. Only Google had a smaller transformer that could somewhat compete (Reformer) but the rest went with the brute force GPT-like models and 1000+ GPUs, $10M+/training run etc. I haven't seen a single post anywhere explaining how to train such large models (a smaller model on 4-8 GPUs doesn't count). "Facts are clear"
    - deleted 4y ago
      [deleted]
    - kir-gadjello 4y ago
      "Accomodate" is the word to scrutinize here. Yes, it will cost a lot to outright buy physical HPC infrastructure to train and infer a series of large models deployed for customers all over the globe. No, it won't cost nearly as much to rent cloud infra to train a similarly-sized model. No, you won't be able to train a large model on a single multi-GPU node, you will need a cluster containing a respectable power of two of GPUs (or other accelerators). It's a widely known meme at this point, but to reiterate: For a popular large model, the largest part of the cost will be spent on inference, not on training. If we assume inference on end user device, this cost disappears. And even if you have the million to rent a cluster, there is a very deep question of the optimal architecture, dataset and hyperparameters to train the best model possible under given constraints.
      - bitL 4y ago
        You are missing the point. Extremely large LLMs don't train the same way as your BERT_Large x8 variety of LLMs. Your whole training procedure is different. Also Microsoft spent so much initially because their Azure Cloud was unable to cope with it electrically and they had to rewire a datacenter for it. So it's not even a question of just renting 1000 GPUs. Do you have actual experience training GPT-3+ sized models?
        kir-gadjello 4y ago
        If you are interested in the infrastructure-level details of how similar models are trained by lesser known groups, take a look at this paper: https://arxiv.org/abs/2204.06745 https://arxiv.org/abs/2204.06745 Quotes from the paper: Our model is trained using a codebase that builds on Megatron (Shoeybi et al., 2020) and DeepSpeed (Rasley et al., 2020) to facilitate efficient and straightforward training of large language models with tens of billions of parameters. We use the official PyTorch v1.10.0 release binary package compiled with CUDA 11.1. This package is bundled with NCCL 2.10.3 for distributed communications. We trained GPT-NeoX-20B on twelve Supermicro AS-4124GO-NART servers, each with eight NVIDIA A100-SXM4-40GB GPUs and configured with two AMD EPYC 7532 CPUs. All GPUs can directly access the InfiniBand switched fabric through one of four ConnectX-6 HCAs for GPUDirect RDMA. Two NVIDIA MQM8700-HS2R switches—connected by 16 links—compose the spine of this InfiniBand network, with one link per node CPU socket connected to each switch. And if you are interested in 176B-scale training, read the BLOOM-176B and OPT-175B papers and research logs.
- redox99 4y ago
  I think a lot of people are going to just ignore the "license".
- oceanplexian 4y ago
  I think anyone can train one of these models. “anyone” being someone who can throw $1-5M in cloud resources at compute. Or a bored sysadmin with access to a few thousand GPUs and a cool boss. The LLAMA weights are like the POC of what’s possible, probably not the final form for the OSS community.
  - bitL 4y ago
    There seems to be a big difference between efficiently training a "large-ish" model on 4-8 GPUs and a gigantic model on 1000+ GPUs. The same techniques might not work due to different warm up steps, gradient overlaps etc. All you can see running in the wild are quantized LLaMA variants (4 or 8-bit) whereas the original model is 32-bit.
  - oezi 4y ago
    Whats possible now (March 2023). The field is moving so quick.
  - wkat4242 4y ago
    Sounds like a great Kickstarter option for a truly free model. I'd back it.
    - kir-gadjello 4y ago
      Please do it, people shouldn't put up with the apathetic siloed status quo. I'm sure people will find all sorts of beneficial uses for these models they are going to run on their own hardware!
      - pdc56 4y ago
        These models are as valuable as code now. And we know how open source eventually took over pretty much all code. I expect this to be the same, but instead of four decades, it will be four months
- azeirah 4y ago
  Is there a reason why we aren't using BLOOM's model? It's huge, 160B parameters. It's open, it's free, it's been out there for a while. Is something wrong with it? Using outdated training data? Using outdated training techniques? Or is it just not well known enough -- ie, not in the news at the right time.
  - kir-gadjello 4y ago
    BLOOM, while being a useful model for underrepresented languages, underperforms openai's GPT-3 models of similar parameter count on english benchmarks. Which is a pity. And OpenAI's known models (at least the ones for which the architecture is publicly known) https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers aren't pinnacles of parameter efficiency either: Deepmind's Chinchilla and LLaMA beat them on benchmarks while being much smaller. Just an opinion, but if we truly aim for "green computing" and carbon footprint reduction, we should pay a lot of attention to the parameter efficiency of the models that get run in our datacenters - and we should invest money into public availability of the most efficient foundation models.
    - mirekrusin 4y ago
      Incentives already exist for being efficient - but of course it doesn't mean less carbon footprint, it means more powerful models. In other words every carbon footprint reduction optimisation is used to yield more efficient output with the same (all available) carbon footprint.
    - espadrine 4y ago
      > underperforms openai's GPT-3 models of similar parameter count on english benchmarks. On the flip side, it overperforms it on every other language. It seems clear to me that it is more useful for popular applications (where a large userbase will use their own tongue). It also has a 7B version. Although it is much weaker than LLaMA, it could be pretrained to match, especially through distillation from LLaMA. By now, it is clear that even Chinchilla is underestimating how much we gain from training more: training is more efficient thanks to new techniques, and even in the paper, some lines are stopped early before reaching Pareto (likely because it ran out of tokens, but LLaMA makes it seem that >1 epoch training is fine).
- generalizations 4y ago
  > Can’t build anything significant with leaked models. If it's behind a SaaS, I think you can.

yieldcrv 4y ago

Feels like we’ll get to “good enough” in a matter of weeks. Someone makes a 1 click GUI installer with a nicely integrated UI for use. No command line use at all. We’ll get model and weight distributors for updated knowledge.

bogwog 4y ago

Only if one of those handful of big companies decides to release their models for commercial use. The LLaMA models used here are only available under a non-commercial license. Can’t build anything significant with leaked models. Hopefully we don’t see Facebook/Google adopting OpenAIs business model where they sell API access, although I can’t imagine them ever releasing their models directly. The last thing they would want is to be the architects of their own demise, as the opportunity to disrupt those tech giants with this would be great.

adam_arthur 4y ago
The techniques to train and reproduce similarly performing models seem fairly widely understood and replicable at this point. I do agree that large orgs are likely to have the best models, but a lot of the low hanging fruit/initial utility is already covered by models that run on consumer hardware
- bitL 4y ago
  Can you post a link on how to train a gigantic LLM on consumer hardware with all detailed steps please? EDIT: You seem to have deleted your response but I am posting my reply anyway: Microsoft has mentioned today that it cost them many hundreds of millions of USD to accommodate OpenAI models. I studied NLP at Stanford including LLMs. Only Google had a smaller transformer that could somewhat compete (Reformer) but the rest went with the brute force GPT-like models and 1000+ GPUs, $10M+/training run etc. I haven't seen a single post anywhere explaining how to train such large models (a smaller model on 4-8 GPUs doesn't count). "Facts are clear"
  - deleted 4y ago
    [deleted]
  - kir-gadjello 4y ago
    "Accomodate" is the word to scrutinize here. Yes, it will cost a lot to outright buy physical HPC infrastructure to train and infer a series of large models deployed for customers all over the globe. No, it won't cost nearly as much to rent cloud infra to train a similarly-sized model. No, you won't be able to train a large model on a single multi-GPU node, you will need a cluster containing a respectable power of two of GPUs (or other accelerators). It's a widely known meme at this point, but to reiterate: For a popular large model, the largest part of the cost will be spent on inference, not on training. If we assume inference on end user device, this cost disappears. And even if you have the million to rent a cluster, there is a very deep question of the optimal architecture, dataset and hyperparameters to train the best model possible under given constraints.
    - bitL 4y ago
      You are missing the point. Extremely large LLMs don't train the same way as your BERT_Large x8 variety of LLMs. Your whole training procedure is different. Also Microsoft spent so much initially because their Azure Cloud was unable to cope with it electrically and they had to rewire a datacenter for it. So it's not even a question of just renting 1000 GPUs. Do you have actual experience training GPT-3+ sized models?
      - kir-gadjello 4y ago
        If you are interested in the infrastructure-level details of how similar models are trained by lesser known groups, take a look at this paper: https://arxiv.org/abs/2204.06745 https://arxiv.org/abs/2204.06745 Quotes from the paper: Our model is trained using a codebase that builds on Megatron (Shoeybi et al., 2020) and DeepSpeed (Rasley et al., 2020) to facilitate efficient and straightforward training of large language models with tens of billions of parameters. We use the official PyTorch v1.10.0 release binary package compiled with CUDA 11.1. This package is bundled with NCCL 2.10.3 for distributed communications. We trained GPT-NeoX-20B on twelve Supermicro AS-4124GO-NART servers, each with eight NVIDIA A100-SXM4-40GB GPUs and configured with two AMD EPYC 7532 CPUs. All GPUs can directly access the InfiniBand switched fabric through one of four ConnectX-6 HCAs for GPUDirect RDMA. Two NVIDIA MQM8700-HS2R switches—connected by 16 links—compose the spine of this InfiniBand network, with one link per node CPU socket connected to each switch. And if you are interested in 176B-scale training, read the BLOOM-176B and OPT-175B papers and research logs.
redox99 4y ago
I think a lot of people are going to just ignore the "license".
oceanplexian 4y ago
I think anyone can train one of these models. “anyone” being someone who can throw $1-5M in cloud resources at compute. Or a bored sysadmin with access to a few thousand GPUs and a cool boss. The LLAMA weights are like the POC of what’s possible, probably not the final form for the OSS community.
- bitL 4y ago
  There seems to be a big difference between efficiently training a "large-ish" model on 4-8 GPUs and a gigantic model on 1000+ GPUs. The same techniques might not work due to different warm up steps, gradient overlaps etc. All you can see running in the wild are quantized LLaMA variants (4 or 8-bit) whereas the original model is 32-bit.
- oezi 4y ago
  Whats possible now (March 2023). The field is moving so quick.
- wkat4242 4y ago
  Sounds like a great Kickstarter option for a truly free model. I'd back it.
  - kir-gadjello 4y ago
    Please do it, people shouldn't put up with the apathetic siloed status quo. I'm sure people will find all sorts of beneficial uses for these models they are going to run on their own hardware!
    - pdc56 4y ago
      These models are as valuable as code now. And we know how open source eventually took over pretty much all code. I expect this to be the same, but instead of four decades, it will be four months
azeirah 4y ago
Is there a reason why we aren't using BLOOM's model? It's huge, 160B parameters. It's open, it's free, it's been out there for a while. Is something wrong with it? Using outdated training data? Using outdated training techniques? Or is it just not well known enough -- ie, not in the news at the right time.
- kir-gadjello 4y ago
  BLOOM, while being a useful model for underrepresented languages, underperforms openai's GPT-3 models of similar parameter count on english benchmarks. Which is a pity. And OpenAI's known models (at least the ones for which the architecture is publicly known) https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers aren't pinnacles of parameter efficiency either: Deepmind's Chinchilla and LLaMA beat them on benchmarks while being much smaller. Just an opinion, but if we truly aim for "green computing" and carbon footprint reduction, we should pay a lot of attention to the parameter efficiency of the models that get run in our datacenters - and we should invest money into public availability of the most efficient foundation models.
  - mirekrusin 4y ago
    Incentives already exist for being efficient - but of course it doesn't mean less carbon footprint, it means more powerful models. In other words every carbon footprint reduction optimisation is used to yield more efficient output with the same (all available) carbon footprint.
  - espadrine 4y ago
    > underperforms openai's GPT-3 models of similar parameter count on english benchmarks. On the flip side, it overperforms it on every other language. It seems clear to me that it is more useful for popular applications (where a large userbase will use their own tongue). It also has a 7B version. Although it is much weaker than LLaMA, it could be pretrained to match, especially through distillation from LLaMA. By now, it is clear that even Chinchilla is underestimating how much we gain from training more: training is more efficient thanks to new techniques, and even in the paper, some lines are stopped early before reaching Pareto (likely because it ran out of tokens, but LLaMA makes it seem that >1 epoch training is fine).
generalizations 4y ago
> Can’t build anything significant with leaked models. If it's behind a SaaS, I think you can.