4 ms·

The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are th

by ftxbro 3y ago

The paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?

FallDead 3y ago
(Note: My comments do not represent or project those of my collaborators) I remember talking to Blink DL about this, I think the plan is just to build an ecosystem, provide more diversity in the DL space. There are plans to make a RWKV5, they are in the open in the RWKV5 channel. From an engineering standpoint I don't really see the "reap" the value of random contributors to the project. Most of us I believe ... are hackers and tinkerers that just want to learn and contribute and be apart of something that can change the current
pico_creator 3y ago
Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not training data, so we compared similarly trained models) And yes we do have multiple experiments and plans to make it better. It’s a list, and we will not know which is final until we try. Individual members can go to great lengths on what they are working on For better or worse, being truly OSS means our initiatives are more disorganized then a centrally planned org
- bluepoint 3y ago
  Can beginners without money with by PhDs, contribute? If so what would be the best way to start?
  - ftxbro 3y ago
    i would guess, go to their discord. also they put their github so you could fix a bug in their github.
  - pico_creator 3y ago
    There are lots of really low hanging fruits - integrating this with AI platform X/Y/Z - setting up evals - improving the code quality - making a how to guide (it’s stuck on my todo list) - helping with dataset - doing silly experiments on how the architecture work (and if the changes give good result) - etc etc One of the community goals is to make this a model for EVERYONE on earth that means we need quality dataset for all the non English languages So even on that level there are things to do ( find something that interest you on the community )
- redox99 3y ago
  > it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board To be fair, that filters the majority of models in the scoreboard.
  - pico_creator 3y ago
    Dun we all wish this wasn’t the case? Where we have more OSS models to choose from without weird rule lawyering gotchas. Or needing to be from a research institute / a license to download the weights

by ftxbro 3y ago

FallDead 3y ago

(Note: My comments do not represent or project those of my collaborators) I remember talking to Blink DL about this, I think the plan is just to build an ecosystem, provide more diversity in the DL space. There are plans to make a RWKV5, they are in the open in the RWKV5 channel. From an engineering standpoint I don't really see the "reap" the value of random contributors to the project. Most of us I believe ... are hackers and tinkerers that just want to learn and contribute and be apart of something that can change the current

pico_creator 3y ago

Completely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not training data, so we compared similarly trained models) And yes we do have multiple experiments and plans to make it better. It’s a list, and we will not know which is final until we try. Individual members can go to great lengths on what they are working on For better or worse, being truly OSS means our initiatives are more disorganized then a centrally planned org

bluepoint 3y ago
Can beginners without money with by PhDs, contribute? If so what would be the best way to start?
- ftxbro 3y ago
  i would guess, go to their discord. also they put their github so you could fix a bug in their github.
- pico_creator 3y ago
  There are lots of really low hanging fruits - integrating this with AI platform X/Y/Z - setting up evals - improving the code quality - making a how to guide (it’s stuck on my todo list) - helping with dataset - doing silly experiments on how the architecture work (and if the changes give good result) - etc etc One of the community goals is to make this a model for EVERYONE on earth that means we need quality dataset for all the non English languages So even on that level there are things to do ( find something that interest you on the community )
redox99 3y ago
> it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board To be fair, that filters the majority of models in the scoreboard.
- pico_creator 3y ago
  Dun we all wish this wasn’t the case? Where we have more OSS models to choose from without weird rule lawyering gotchas. Or needing to be from a research institute / a license to download the weights