5 ms·

Jesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.

lordofgibbons 3y ago
They're a group of volunteers who came together over Discord.
ftxbro 3y ago
I think it's just, everyone in their discord channel.
- Buttons840 3y ago
  What channel? I have an application for sequence models I think might be novel, and I'd like to be able to get credit and help research it if possible. Probably somebody has already done it, but I cannot search well enough to find related literature.
  - ftxbro 3y ago
    Write it and submit it yourself. Sole authorship is given more weight these days for whatever reason. In a multi author publication, even if you are first author or listed as equal contribution, if there is a more famous person on the paper everyone will assume they did it.
    - Buttons840 3y ago
      Write it and submit it to who? I'm not familiar enough with the field to find any prior work or related work.
      - ftxbro 3y ago
        find the most closely related paper that you know of, even if it's not very close, and submit your idea to the same journal that published that other idea
  - rahimnathwani 3y ago
    > What channel? https://discord.gg/qt9egFA7ve https://discord.gg/qt9egFA7ve
- stellaathena 3y ago
  Our discord servers (a primary one, a spin-off for RWKV, another spinoff for BioML, etc) have tens of thousands of people between them :) So not quite everyone. But this was a community effort with a public call for contributions
  - deleted 3y ago
    [deleted]
  - ftxbro 3y ago
    Are you the Stella from the Acknowledgements section? Are you in one of those spinoffs like BioML, or you aren't on the author list because you have some conflict with your work where you can't legally do something because of intellectual property laws or license things or NDA? Also when you talk about 'our discord servers' which ones do you mean? Is it ones run by Eleuther?
    - stellaathena 3y ago
      Yes I am (apparently) in the acknowledgments. I was not an author on the paper because I didn’t have time to contribute too much. I also try to err on the side of not being added to papers, as my position (I run EleutherAI) tends to encourage people to be overgenerous with offers. I anticipate having more time this coming month and being on the version that’s submitted for peer review, but we’ll see. BlinkDL has been working on this project for two-ish years, originally in the EleutherAI discord and then created his own to house the project. I wasn’t thinking too hard about my exact wording, but yes I was thinking of EleutherAI and its various spin-off servers. EleutherAI doesn’t /run/ any of the other servers, but we all have a close collaborative relationship. I’m sure there’s a lot of duplication of membership (e.g., I’m in all of them) but quickly adding up the membership of each server comes out to around 70,000. EleutherAI and LAION are the largest at 25k each, with the others typically having around 5k each. I would expect at least 30k of those users to be unique though.
cttet 3y ago
The model is created by one person, he does not have enough time to write the paper
pico_creator 3y ago
RWKV has been running in public (relatively obscure to other models) for the past 2 years Mostly lead by a single person (blink). This community consist mostly of people outside the academia / big VC tech scene When eleutherAI offered to help us with writing the paper. Various key folks banded together for the paper, as it’s what seems to be a very strong alternative to transformers This does not mean everyone in the discord was credited. The requirements are for significant contributions to the paper. typically several paragraphs long worth of drafting and revisions Just doing a line of grammar change or a single benchmark is not enough
- pico_creator 3y ago
  AKA: no one here is incentivised to fight for bigger ownership, no promotion or KPI or pay was on the line
evolve7942 3y ago
If you think this is wild, see the PaLM 2 paper with 2.5 pages of 2 column attributions. https://arxiv.org/pdf/2305.10403.pdf https://arxiv.org/pdf/2305.10403.pdf
- quickthrower2 3y ago
  Page 28. That is wild, almost like film credits
- visarga 3y ago
  No, the Bloom paper is wild: 2.5 pages of author names written with no spacing, I counted 472 authors after deduplication. PaLM 2 has only 181 authors. ;) https://arxiv.org/pdf/2211.05100.pdf https://arxiv.org/pdf/2211.05100.pdf
skykooler 3y ago
It's Type 10 of https://xkcd.com/2456/ https://xkcd.com/2456/.
- pico_creator 3y ago
  Hahaha. True. And even then it’s an undersell (the limited scope of the paper drops lots of the things being done)
koningrobot 3y ago
Have you seen the author list of DeepSpeech2? https://arxiv.org/abs/1512.02595 https://arxiv.org/abs/1512.02595

lordofgibbons 3y ago

They're a group of volunteers who came together over Discord.

ftxbro 3y ago

I think it's just, everyone in their discord channel.

Buttons840 3y ago
What channel? I have an application for sequence models I think might be novel, and I'd like to be able to get credit and help research it if possible. Probably somebody has already done it, but I cannot search well enough to find related literature.
- ftxbro 3y ago
  Write it and submit it yourself. Sole authorship is given more weight these days for whatever reason. In a multi author publication, even if you are first author or listed as equal contribution, if there is a more famous person on the paper everyone will assume they did it.
  - Buttons840 3y ago
    Write it and submit it to who? I'm not familiar enough with the field to find any prior work or related work.
    - ftxbro 3y ago
      find the most closely related paper that you know of, even if it's not very close, and submit your idea to the same journal that published that other idea
- rahimnathwani 3y ago
  > What channel? https://discord.gg/qt9egFA7ve https://discord.gg/qt9egFA7ve
stellaathena 3y ago
Our discord servers (a primary one, a spin-off for RWKV, another spinoff for BioML, etc) have tens of thousands of people between them :) So not quite everyone. But this was a community effort with a public call for contributions
- deleted 3y ago
  [deleted]
- ftxbro 3y ago
  Are you the Stella from the Acknowledgements section? Are you in one of those spinoffs like BioML, or you aren't on the author list because you have some conflict with your work where you can't legally do something because of intellectual property laws or license things or NDA? Also when you talk about 'our discord servers' which ones do you mean? Is it ones run by Eleuther?
  - stellaathena 3y ago
    Yes I am (apparently) in the acknowledgments. I was not an author on the paper because I didn’t have time to contribute too much. I also try to err on the side of not being added to papers, as my position (I run EleutherAI) tends to encourage people to be overgenerous with offers. I anticipate having more time this coming month and being on the version that’s submitted for peer review, but we’ll see. BlinkDL has been working on this project for two-ish years, originally in the EleutherAI discord and then created his own to house the project. I wasn’t thinking too hard about my exact wording, but yes I was thinking of EleutherAI and its various spin-off servers. EleutherAI doesn’t /run/ any of the other servers, but we all have a close collaborative relationship. I’m sure there’s a lot of duplication of membership (e.g., I’m in all of them) but quickly adding up the membership of each server comes out to around 70,000. EleutherAI and LAION are the largest at 25k each, with the others typically having around 5k each. I would expect at least 30k of those users to be unique though.

cttet 3y ago

The model is created by one person, he does not have enough time to write the paper

pico_creator 3y ago

RWKV has been running in public (relatively obscure to other models) for the past 2 years Mostly lead by a single person (blink). This community consist mostly of people outside the academia / big VC tech scene When eleutherAI offered to help us with writing the paper. Various key folks banded together for the paper, as it’s what seems to be a very strong alternative to transformers This does not mean everyone in the discord was credited. The requirements are for significant contributions to the paper. typically several paragraphs long worth of drafting and revisions Just doing a line of grammar change or a single benchmark is not enough

pico_creator 3y ago
AKA: no one here is incentivised to fight for bigger ownership, no promotion or KPI or pay was on the line

evolve7942 3y ago

If you think this is wild, see the PaLM 2 paper with 2.5 pages of 2 column attributions. https://arxiv.org/pdf/2305.10403.pdf https://arxiv.org/pdf/2305.10403.pdf

quickthrower2 3y ago
Page 28. That is wild, almost like film credits
visarga 3y ago
No, the Bloom paper is wild: 2.5 pages of author names written with no spacing, I counted 472 authors after deduplication. PaLM 2 has only 181 authors. ;) https://arxiv.org/pdf/2211.05100.pdf https://arxiv.org/pdf/2211.05100.pdf

skykooler 3y ago

It's Type 10 of https://xkcd.com/2456/ https://xkcd.com/2456/.

pico_creator 3y ago
Hahaha. True. And even then it’s an undersell (the limited scope of the paper drops lots of the things being done)

koningrobot 3y ago

Have you seen the author list of DeepSpeech2? https://arxiv.org/abs/1512.02595 https://arxiv.org/abs/1512.02595