4 ms·

So is the next step is for someone to come in a fine tune on top of it in order to make it a Vicuna? Or can current vicuna deltas be applied?

rafaelero 3y ago
Yeah, it's pretty trivial to change the base model from LLaMa to this next one. You just have to finetune it with the same data used previously to train Vicuna.
wesleychen 3y ago
There's no model yet, only a dataset.
- omneity 3y ago
  My understanding is that LLaMa's architecture is open, so the most difficult part is: 1. Getting data of equal or better quality 2. Securing the funding/hardware required for training 3. Learning/figuring out the training challenges needed to tune the process (the PhD part) It seems #1 is the relatively lowest hanging fruit and a prerequisite for the other two, and that's what the project is (rightfully) tackling at this stage. #2 could be solved by many ways, and doesn't require much innovation if the project and the team are solid. Which takes me to #3, which on the other hand seems to be the make or break part of the project. I'm not one to doubt the technical prowesses of the RedPajama's team and their contributors, I rather see it economically. How can an AI open-source project compete with big tech in attracting the brilliant minds of our generation? It's enough to look at levels.xyz to see the battle is not ... level. There's a serious economical challenge in here to have any sort of sustainable open source initiative in AI.
jamiedg 3y ago
Hi! I lead Product at Together. We will be releasing a full suite of models trained on this data starting with the first models in the coming weeks. We will release RedPajama base models and RedPajama instruction-tuned models. All of the models will be released under the Apache 2.0 license, allowing commercial use. Therefore, anyone will be able to fine-tune the RedPajama models using Vicuna or other datasets, given they will be fully open-source. The RedPajama instruction-tuned models will be fine-tuned only with instruction labels from human labelers and OpenChatKit feedback (). We feel this will keep these models fully "clean" for use in commercial applications without using the output of other commercial models like were used in Alpaca or Vicuna. However, we'll be excited to see all the great fine-tunes created by the open community and are eager to see how close open-source models can get to the quality of leading commercial models over time!! () OpenChatKit: https://huggingface.co/spaces/togethercomputer/OpenChatKit https://huggingface.co/spaces/togethercomputer/OpenChatKit

rafaelero 3y ago

Yeah, it's pretty trivial to change the base model from LLaMa to this next one. You just have to finetune it with the same data used previously to train Vicuna.

wesleychen 3y ago

There's no model yet, only a dataset.

omneity 3y ago
My understanding is that LLaMa's architecture is open, so the most difficult part is: 1. Getting data of equal or better quality 2. Securing the funding/hardware required for training 3. Learning/figuring out the training challenges needed to tune the process (the PhD part) It seems #1 is the relatively lowest hanging fruit and a prerequisite for the other two, and that's what the project is (rightfully) tackling at this stage. #2 could be solved by many ways, and doesn't require much innovation if the project and the team are solid. Which takes me to #3, which on the other hand seems to be the make or break part of the project. I'm not one to doubt the technical prowesses of the RedPajama's team and their contributors, I rather see it economically. How can an AI open-source project compete with big tech in attracting the brilliant minds of our generation? It's enough to look at levels.xyz to see the battle is not ... level. There's a serious economical challenge in here to have any sort of sustainable open source initiative in AI.

jamiedg 3y ago

Hi! I lead Product at Together. We will be releasing a full suite of models trained on this data starting with the first models in the coming weeks. We will release RedPajama base models and RedPajama instruction-tuned models. All of the models will be released under the Apache 2.0 license, allowing commercial use. Therefore, anyone will be able to fine-tune the RedPajama models using Vicuna or other datasets, given they will be fully open-source. The RedPajama instruction-tuned models will be fine-tuned only with instruction labels from human labelers and OpenChatKit feedback (). We feel this will keep these models fully "clean" for use in commercial applications without using the output of other commercial models like were used in Alpaca or Vicuna. However, we'll be excited to see all the great fine-tunes created by the open community and are eager to see how close open-source models can get to the quality of leading commercial models over time!! () OpenChatKit: https://huggingface.co/spaces/togethercomputer/OpenChatKit https://huggingface.co/spaces/togethercomputer/OpenChatKit