5 ms·

Open-Llama: Complete training pipeline for building large language models

141 points by bayes-song 3y ago

marcopicentini 3y ago
Sorry for newbie question. What’s the advantage of retraining the model versus using an already trained model through API (OPEN AI). I understand the economic principle of “make or buy”, but is there else?
- bayes-song 3y ago
  This is a good question. Personally, I think it is a better way for most companies to directly use the API of OpenAI, which is currently the best in terms of ease of use and ability in the general field. I think there are only a few types that need to retrain the model themselves. 1. Researchers who are very cutting-edge and have sufficient funds. 2. Large companies that require ultra large amounts of calls or have strict confidentiality requirements. 3. Application scenarios in very specific fields, such as chemical molecules, may require more domain specific pretraining.
  - marcopicentini 3y ago
    The problem is that making a summary of a text of 100k token it costs 2$ using Davinci.
bayes-song 3y ago
Check out this model trained using the Open-Llama project at http://home.ustc.edu.cn/~sl9292 http://home.ustc.edu.cn/~sl9292 . This model is trained primarily on English and Chinese, but also has capabilities in other languages like Japanese and Korean. Now, let's dive into Open-Llama. It's a truly open-source project for pre-training and instruct-tuning AI models. One of the key features of this project is its support for a wide range of model sizes, from 7B to 65B parameters. What sets Open-Llama apart is the incorporation of performance acceleration via xformers from Llama, enabling 95% of the original Llama speed on the 65B models. In fact, for the 7B models, Open-Llama's performance surpasses the original Llama. By providing full access to the codebase, we believe that Open-Llama will contribute greatly to the advancement of open-source AI technologies. We invite developers and researchers to join us on this exciting journey!
lhl 3y ago
Namespace collisions are inevitable, especially w/ how fast-moving the LLM space is right now, just wanted to point out that besides this "Open-Llama" project (which looks really interesting, and well documented in the Github repo), there is also another group training "OpenLLaMA" https://github.com/openlm-research/open_llama https://github.com/openlm-research/open_llama (which looks like an effort by two Berkeley PhD students, https://www.haoliu.site/ https://www.haoliu.site/ and http://young-geng.xyz/ http://young-geng.xyz/ to reproduce LLaMA using the 1.2T token Together RedPajama dataset. They've released up to a 300B checkpoint so far.) Feedback for /u/bayes-song - it'd be great to have a more info on the model card on HF - right now it's unclear the parameter count, # of total tokens you're planning on training on/how many you've trained on so far. An Evaluation section (maybe using lm-evaluation-harness) might be good as well?
- bayes-song 3y ago
  Thank you for your suggestion, this will indeed be more intuitive, I will add relevant results as soon as possible.
- pythux 3y ago
  To add to that, I believe the title of this submission is a reference ("Open-Lamam: A “real” open-source project to train LLM not just checkpoints") to this project you link, since they did not (to my knowledge), release the code for the training or detailed instructions to reproduce their experiment precisely, only checkpoints.
christkv 3y ago
Feels like its still the area of wait and see as the space shakes out. It would be great to be able to run our own models in some near future for applications but the amount of hardware needed to delivery service to a significant audience is pretty crazy. Right now I don't see any way but to re-bill the cost with a markup to end customers unless you have a giant pile of VC money that you can light on fire.
- quickthrower2 3y ago
  I found running the model on rented hardware much more expensive than ChatGPT. Might work ok for local sunk cost hardware for those who game and don’t crypto mine.
- ingenieroariel 3y ago
  WebGPU would be a way to shift that cost back to each device. RedPajama 3B could become useful for some tasks and run quite fast on most hardware available. Then as users have better computers, they can get access to better models?
  - snovv_crash 3y ago
    You'd have to have each user download a 3GB payload first. For comparison, that's a good few hours of netflix at 1080p.
    - sroussey 3y ago
      Browsers and operating systems will eventually include them. But for now, you would need a good privacy reason to go this route.
deleted 3y ago
[deleted]

141 points by bayes-song 3y ago

Sorry for newbie question. What’s the advantage of retraining the model versus using an already trained model through API (OPEN AI). I understand the economic principle of “make or buy”, but is there else?

bayes-song 3y ago
This is a good question. Personally, I think it is a better way for most companies to directly use the API of OpenAI, which is currently the best in terms of ease of use and ability in the general field. I think there are only a few types that need to retrain the model themselves. 1. Researchers who are very cutting-edge and have sufficient funds. 2. Large companies that require ultra large amounts of calls or have strict confidentiality requirements. 3. Application scenarios in very specific fields, such as chemical molecules, may require more domain specific pretraining.
- marcopicentini 3y ago
  The problem is that making a summary of a text of 100k token it costs 2$ using Davinci.

bayes-song 3y ago

Check out this model trained using the Open-Llama project at http://home.ustc.edu.cn/~sl9292 http://home.ustc.edu.cn/~sl9292 . This model is trained primarily on English and Chinese, but also has capabilities in other languages like Japanese and Korean. Now, let's dive into Open-Llama. It's a truly open-source project for pre-training and instruct-tuning AI models. One of the key features of this project is its support for a wide range of model sizes, from 7B to 65B parameters. What sets Open-Llama apart is the incorporation of performance acceleration via xformers from Llama, enabling 95% of the original Llama speed on the 65B models. In fact, for the 7B models, Open-Llama's performance surpasses the original Llama. By providing full access to the codebase, we believe that Open-Llama will contribute greatly to the advancement of open-source AI technologies. We invite developers and researchers to join us on this exciting journey!

lhl 3y ago

Namespace collisions are inevitable, especially w/ how fast-moving the LLM space is right now, just wanted to point out that besides this "Open-Llama" project (which looks really interesting, and well documented in the Github repo), there is also another group training "OpenLLaMA" https://github.com/openlm-research/open_llama https://github.com/openlm-research/open_llama (which looks like an effort by two Berkeley PhD students, https://www.haoliu.site/ https://www.haoliu.site/ and http://young-geng.xyz/ http://young-geng.xyz/ to reproduce LLaMA using the 1.2T token Together RedPajama dataset. They've released up to a 300B checkpoint so far.) Feedback for /u/bayes-song - it'd be great to have a more info on the model card on HF - right now it's unclear the parameter count, # of total tokens you're planning on training on/how many you've trained on so far. An Evaluation section (maybe using lm-evaluation-harness) might be good as well?

bayes-song 3y ago
Thank you for your suggestion, this will indeed be more intuitive, I will add relevant results as soon as possible.
pythux 3y ago
To add to that, I believe the title of this submission is a reference ("Open-Lamam: A “real” open-source project to train LLM not just checkpoints") to this project you link, since they did not (to my knowledge), release the code for the training or detailed instructions to reproduce their experiment precisely, only checkpoints.

christkv 3y ago

Feels like its still the area of wait and see as the space shakes out. It would be great to be able to run our own models in some near future for applications but the amount of hardware needed to delivery service to a significant audience is pretty crazy. Right now I don't see any way but to re-bill the cost with a markup to end customers unless you have a giant pile of VC money that you can light on fire.

quickthrower2 3y ago
I found running the model on rented hardware much more expensive than ChatGPT. Might work ok for local sunk cost hardware for those who game and don’t crypto mine.
ingenieroariel 3y ago
WebGPU would be a way to shift that cost back to each device. RedPajama 3B could become useful for some tasks and run quite fast on most hardware available. Then as users have better computers, they can get access to better models?
- snovv_crash 3y ago
  You'd have to have each user download a 3GB payload first. For comparison, that's a good few hours of netflix at 1080p.
  - sroussey 3y ago
    Browsers and operating systems will eventually include them. But for now, you would need a good privacy reason to go this route.

deleted 3y ago

[deleted]