3 ms·

Is there a truly open source effort in the LLM space? Like a collaborative, crowd-sourced effort (possibly with academic institutions playing a major role) that

Is there a truly open source effort in the LLM space? Like a collaborative, crowd-sourced effort (possibly with academic institutions playing a major role) that relies on creative commons licensed or otherwise open data and produces a public good as final outcome? There is this ridiculous idea of AI moats and other machinations for the next big VC thing (god bless them, people have spend their energy on worse pursuits) but in a fundamental sense there is a public good type infrastructure crying out to be developed for each major linguistic domain. Maybe such an effort would not be cutting edge enough to power the next corporate chatbot that will eliminate 99% of all jobs, but it would be a significant step up in our ability to process text.

vinni2 3y ago
I think OpenAssistant is the closest to what you are describing but their models are not yet that great. https://open-assistant.io/ https://open-assistant.io/
- nulld3v 3y ago
  Open Assistant just shut down: https://www.youtube.com/watch?v=gqtmUHhaplo https://www.youtube.com/watch?v=gqtmUHhaplo Cited reasons: Lack of resources, lack of maintainer time and there being many new good alternatives.
  - vinni2 3y ago
    oh I didn’t know that.
dartos 3y ago
RWKV is fully open source and even part of the Linux foundation Idk why nobody ever talks about it
TheCaptain4815 3y ago
ElutherAi fits that I believe. In the olden days (1.5 years ago) they probably had the best open source model with their NeoX model, but it’s been ellipsed by Llamma and other “open source” models I believe. They still have an active discord with a great community pushing forward.
emadm 3y ago
We back rwkv, eleuther ai and others at stability ai We also have our carper.ai lab for the rl buts We are rolling out open language models and datasets soon for a number of languages too, see our recent Japanese language models for example Got some big plans soon, have funded it all ourself but sure other would like to help
- senseiV 3y ago
  ah yes RWKV, always great to mention, crazy about how no one talks about it, it literally the most powerful multilang model at 1b and 3b scales, probs going for 14b and 7b too
doctorpangloss 3y ago
> that relies on creative commons licensed or otherwise open data You can try very hard to make neural network stuff a holistic social experience. There is a lot of value in that! I think it's meaningless though, a colossal waste of time. In the objective reality we live in: We wouldn't be talking about transformers, attention, etc. if it weren't for papers that used so called "not" "open data." It's all tainted. There's no shortcuts. If you buy into holistic social experiences as an essential part of your chatbot or whatever, you expose yourself to being sniped in some basic way by merely one comment on the Internet. Bullshit Street is a two lane road.

vinni2 3y ago

I think OpenAssistant is the closest to what you are describing but their models are not yet that great. https://open-assistant.io/ https://open-assistant.io/

nulld3v 3y ago
Open Assistant just shut down: https://www.youtube.com/watch?v=gqtmUHhaplo https://www.youtube.com/watch?v=gqtmUHhaplo Cited reasons: Lack of resources, lack of maintainer time and there being many new good alternatives.
- vinni2 3y ago
  oh I didn’t know that.

dartos 3y ago

RWKV is fully open source and even part of the Linux foundation Idk why nobody ever talks about it

TheCaptain4815 3y ago

ElutherAi fits that I believe. In the olden days (1.5 years ago) they probably had the best open source model with their NeoX model, but it’s been ellipsed by Llamma and other “open source” models I believe. They still have an active discord with a great community pushing forward.

emadm 3y ago

We back rwkv, eleuther ai and others at stability ai We also have our carper.ai lab for the rl buts We are rolling out open language models and datasets soon for a number of languages too, see our recent Japanese language models for example Got some big plans soon, have funded it all ourself but sure other would like to help

senseiV 3y ago
ah yes RWKV, always great to mention, crazy about how no one talks about it, it literally the most powerful multilang model at 1b and 3b scales, probs going for 14b and 7b too

doctorpangloss 3y ago

> that relies on creative commons licensed or otherwise open data You can try very hard to make neural network stuff a holistic social experience. There is a lot of value in that! I think it's meaningless though, a colossal waste of time. In the objective reality we live in: We wouldn't be talking about transformers, attention, etc. if it weren't for papers that used so called "not" "open data." It's all tainted. There's no shortcuts. If you buy into holistic social experiences as an essential part of your chatbot or whatever, you expose yourself to being sniped in some basic way by merely one comment on the Internet. Bullshit Street is a two lane road.