6 ms·

Please test their models before you take it at face value. Eleuther has a history of claiming to replicate projects when they haven't. For example, they shippe

Please test their models before you take it at face value. Eleuther has a history of claiming to replicate projects when they haven't. For example, they shipped a DALL-E repo a few days after OpenAI announced it (https://twitter.com/theshawwn/status/1348017515897659392 https://twitter.com/theshawwn/status/1348017515897659392) which was broken, and they've walked back their GPT-3 replication claims to replicating 1.5B due to the fact that their architecture doesn't scale. As far as I can tell, they're generating a large amount of hype with grandiose claims that they can't deliver on. All I care about is whether you like their models and actually use them in practice. If you do, please let me know and I'll pipe down. But so far, I haven't heard of anyone who uses anything they've produced, and that worries me. Has anyone? One specific claim they made: https://twitter.com/BlancheMinerva/status/1347276975547809809 https://twitter.com/BlancheMinerva/status/134727697554780980... "DALL-E is quite straight forward and already coded. We just need data to train it." No, DALL-E is neither straightforward nor was it successfully coded, especially back on January 7th. Anyway, carry on. I really don't like speaking badly of AI projects, and I hope that they succeed. The model release today is a good step forward, assuming it works. But it might be better to have the expectation of "the models don't work" until proven otherwise. I'd also like to point out that there are some capable people doing work at Eleuther. Sid in particular is one of the best TPU hackers in the scene. I just wish they would scale down their claims, release more models, and not claim that they've done X until actually doing X. For example, the readme says they have "the ability to scale up to full GPT3 sizes (and possibly more!), using the mesh-tensorflow library," which they don't.

natch 6y ago
Good perspective but I would like to hear the response from the developers before concluding too much. This is not meant as a goad to you, but more just as more info for everyone, my understanding is it is an open source community of like minded people type project (as opposed to a bigco) and actively solicits contributions (by which I mean code and data) so anyone seeing room for improvement is welcome to step in from what I can tell. I did find your comment helpful and informative; just adding another angle here.
- stellaathena 6y ago
  It's literally a couple people hanging out in a discord channel and doing this as a way to procrastinate their jobs.
  - mirekrusin 6y ago
    Peak of laziness - build ai to do your job so you have more time building ai.
  - ShamelessC 6y ago
    I think it would help your PR efforts to let people know that more often. People hear "we are <org_name>" and assume you're, you know, an organization. That comes with some amount of expected bug fixes, documentation, verifying results _before_ you release, etc. I'm not really sure how much you gain by attracting tons of people to the discord if you finally release and everyone has unreasonable expectations due to the way you advertise the group as a whole.
    - stellaathena 6y ago
      This'll probably sound silly, but we weren't really expecting much of a response. We've been sitting on these models (they were trained for the Pile paper) for months for no particular reason besides being focused on other things. We figured we'd put it out there in case anyone was interested and then Aran's tweet blew up hard.
      - ShamelessC 6y ago
        Fair enough! I can personally attest to the results being quite impressive now. Fantastic work and apologies for the criticism. It must be a strange situation to get dragged in to all this due to a viral tweet.
ImprobableTruth 6y ago
Disclaimer: I know absolutely nothing about machine learning. Isn't GPT-3 the architecture? Are they doing something different or why would it not scale?
- m00x 6y ago
  It's the model, not the architecture, but you could say the model contains the architecture.
- nmfisher 6y ago
  GPT-3 is the name for the architecture, but there are a few different versions/sizes. The OpenAI version that impressed us all was ~170B parameters, this is far smaller. To go from 2.7B to 170B parameters will need more than just a few config tweaks. There's a whole bunch of hacks and tricks needed to coax a model to train at that scale, the Eleuther version is almost guaranteed to fail out-of-the-box.
  - minimaxir 6y ago
    It's worth noting that the GPT-3 paper did train models with more sane sizes (e.g. 1.5B) as a point of comparison. I am surprised/annoyed they never released them though.
    - stellaathena 6y ago
      It's because OpenAI sells them for profit. The "Ada" model is the same size as the larger of these two EleutherAI models.
      - minimaxir 6y ago
        Huh, I was wondering what the size of the non-davinci models were; guess that make sense. It's still telling that a "small" GPT-3 model can risk cannibalizing a larger model.
        6y ago
cookiengineer 6y ago
What I find interesting about their marketing(?) is that they identified a market niche that they want to position themselves in. Enterprise customers that have no idea about the technical details will just hear about OpenAI's success in this fancy new model and assume that Eleuther can deliver. I mean, most use cases for "big data" projects that are tiny in comparison with Alphabet's datasets will just work with GPT2 fine, probably. And Enterprise customers that hear those claims and see some code, maybe some demo, is enough for them to start the consultancy process. In my opinion that's a policy problem that OpenAI introduced by not requiring the absolute reproducability of both the code and model, and both training and dataset of their models upon release. Stakes are pretty high in the AI industry, and OpenAI actively influences it. My dream was in the beginning that they are a source of verification, audits and "proof" that models are legit...yet I have the feeling lately that they just buzz around like everyone else. To this date I haven't seen anyone replicate any of the DNC results, for example. Anyways, just my two cents on this one.
- leogao 6y ago
  To date, EleutherAI as an "organization" (read: basically a Discord server) has not really attempted any kind of marketing. It has no PR dept, just individuals tweeting about the work that Eleuther does.
  - ma2rten 6y ago
    Also I'm pretty sure there is no consultancy process.
    - deleted 6y ago
      [deleted]
loxias 6y ago
Geez, that's really harsh. I don't think any single thing you've claimed is factually wrong, and I don't speak for Eleuther nor am I attempting to justify their claims. But. As I understand it (mostly from lurking on their discord and reading publicly available materials) this is a group of volunteer academic types trying to replicate something great and awesome, with the only goal of giving it to the world. You could cut them some slack. I can't speak for you, but as a "for free, weekend project" what they've done certainly makes me feel I need to up my game.
- OgAstorga 6y ago
  This has nothing to do with the good work, awesome intentions, nor the fact that they have no financial incentives behind this. Claiming something that is not true is in itself wrong.
  - deleted 6y ago
    [deleted]
  - stellaathena 6y ago
    I am sorry that I was misinformed about the state of our DALL-E replication when I made that tweet. It was not malicious - I was reporting what I had been told by someone else. Yes I was wrong. That said, I had hoped that maybe after two an a half months Shawn would stop holding it over my head.
  - loxias 6y ago
    > Claiming something that is not true is in itself wrong. I 100% agree with this. I also think that one catches more flies with honey than vinegar, and the criticism in the parent comment, while possibly valid, could be phrased more encouragingly and less combatively. It's easy to criticize, it's hard to create, and it's even harder to release.
  - wiz21c 6y ago
    > Claiming something that is not true is in itself wrong. yup, in any project, and especially the one done for the community where the only you get is satisfaction and fame, the success is super tied to communication. Good, honest communication is what builds trust.
nullc 6y ago
Wouldn't it be nice if OpenAI were like .. actually open? :P
- You-Are-Right 6y ago
  Freedom is Slavery.
- ryanackley 6y ago
  Not just that. To even get access to their API, you need to apply. That is the future of AI I'm afraid without projects like this. That is Elites controlling AI and deciding who is "worthy" to use it. I'm sure they have the best of intentions but "worthiness" is subjective.
  - TremendousJudge 6y ago
    Depends on who you mean by "they". If you mean the researchers, then sure, they probably actually believe whatever's written in their ethics statement. Now, the actual owners? I don't believe it for a second
- brunoluiz 6y ago
  Considering last decade social consequences due to easy access to APIs and data, I am quite happy that these initiatives are cautious around opening up software which can have huge impact on society.
  - hesdeadjim 6y ago
    Unfortunately, the cat is out of the bag. Their methods are documented and the results exciting, so to a bad actor (especially state-sponsored) it's completely justified to spend millions attempting to replicate their results from what is publicly available.
deleted 6y ago
[deleted]
6gvONxR4sf7o 6y ago
People are always claiming to release replicated models by replicating the architecture (or main parts of it) but not testing whether it produces the same level of results. It's maddening, especially when the level of results is so directly measurable (just measure what the paper did, not that it's easy, just concrete).
- stellaathena 6y ago
  Our README has comparisons with GPT-2 and GPT-3
  - 6gvONxR4sf7o 6y ago
    Is there anything on its few shot learning performance? I took few shot learning as the main point of GPT-3. Sorry if I just overlooked it, but I don’t see anything on few shot learning in the readme.

natch 6y ago

Good perspective but I would like to hear the response from the developers before concluding too much. This is not meant as a goad to you, but more just as more info for everyone, my understanding is it is an open source community of like minded people type project (as opposed to a bigco) and actively solicits contributions (by which I mean code and data) so anyone seeing room for improvement is welcome to step in from what I can tell. I did find your comment helpful and informative; just adding another angle here.

stellaathena 6y ago
It's literally a couple people hanging out in a discord channel and doing this as a way to procrastinate their jobs.
- mirekrusin 6y ago
  Peak of laziness - build ai to do your job so you have more time building ai.
- ShamelessC 6y ago
  I think it would help your PR efforts to let people know that more often. People hear "we are <org_name>" and assume you're, you know, an organization. That comes with some amount of expected bug fixes, documentation, verifying results _before_ you release, etc. I'm not really sure how much you gain by attracting tons of people to the discord if you finally release and everyone has unreasonable expectations due to the way you advertise the group as a whole.
  - stellaathena 6y ago
    This'll probably sound silly, but we weren't really expecting much of a response. We've been sitting on these models (they were trained for the Pile paper) for months for no particular reason besides being focused on other things. We figured we'd put it out there in case anyone was interested and then Aran's tweet blew up hard.
    - ShamelessC 6y ago
      Fair enough! I can personally attest to the results being quite impressive now. Fantastic work and apologies for the criticism. It must be a strange situation to get dragged in to all this due to a viral tweet.

ImprobableTruth 6y ago

Disclaimer: I know absolutely nothing about machine learning. Isn't GPT-3 the architecture? Are they doing something different or why would it not scale?

m00x 6y ago
It's the model, not the architecture, but you could say the model contains the architecture.
nmfisher 6y ago
GPT-3 is the name for the architecture, but there are a few different versions/sizes. The OpenAI version that impressed us all was ~170B parameters, this is far smaller. To go from 2.7B to 170B parameters will need more than just a few config tweaks. There's a whole bunch of hacks and tricks needed to coax a model to train at that scale, the Eleuther version is almost guaranteed to fail out-of-the-box.
- minimaxir 6y ago
  It's worth noting that the GPT-3 paper did train models with more sane sizes (e.g. 1.5B) as a point of comparison. I am surprised/annoyed they never released them though.
  - stellaathena 6y ago
    It's because OpenAI sells them for profit. The "Ada" model is the same size as the larger of these two EleutherAI models.
    - minimaxir 6y ago
      Huh, I was wondering what the size of the non-davinci models were; guess that make sense. It's still telling that a "small" GPT-3 model can risk cannibalizing a larger model.
      - 6y ago

cookiengineer 6y ago

What I find interesting about their marketing(?) is that they identified a market niche that they want to position themselves in. Enterprise customers that have no idea about the technical details will just hear about OpenAI's success in this fancy new model and assume that Eleuther can deliver. I mean, most use cases for "big data" projects that are tiny in comparison with Alphabet's datasets will just work with GPT2 fine, probably. And Enterprise customers that hear those claims and see some code, maybe some demo, is enough for them to start the consultancy process. In my opinion that's a policy problem that OpenAI introduced by not requiring the absolute reproducability of both the code and model, and both training and dataset of their models upon release. Stakes are pretty high in the AI industry, and OpenAI actively influences it. My dream was in the beginning that they are a source of verification, audits and "proof" that models are legit...yet I have the feeling lately that they just buzz around like everyone else. To this date I haven't seen anyone replicate any of the DNC results, for example. Anyways, just my two cents on this one.

leogao 6y ago
To date, EleutherAI as an "organization" (read: basically a Discord server) has not really attempted any kind of marketing. It has no PR dept, just individuals tweeting about the work that Eleuther does.
- ma2rten 6y ago
  Also I'm pretty sure there is no consultancy process.
  - deleted 6y ago
    [deleted]

loxias 6y ago

Geez, that's really harsh. I don't think any single thing you've claimed is factually wrong, and I don't speak for Eleuther nor am I attempting to justify their claims. But. As I understand it (mostly from lurking on their discord and reading publicly available materials) this is a group of volunteer academic types trying to replicate something great and awesome, with the only goal of giving it to the world. You could cut them some slack. I can't speak for you, but as a "for free, weekend project" what they've done certainly makes me feel I need to up my game.

OgAstorga 6y ago
This has nothing to do with the good work, awesome intentions, nor the fact that they have no financial incentives behind this. Claiming something that is not true is in itself wrong.
- deleted 6y ago
  [deleted]
- stellaathena 6y ago
  I am sorry that I was misinformed about the state of our DALL-E replication when I made that tweet. It was not malicious - I was reporting what I had been told by someone else. Yes I was wrong. That said, I had hoped that maybe after two an a half months Shawn would stop holding it over my head.
- loxias 6y ago
  > Claiming something that is not true is in itself wrong. I 100% agree with this. I also think that one catches more flies with honey than vinegar, and the criticism in the parent comment, while possibly valid, could be phrased more encouragingly and less combatively. It's easy to criticize, it's hard to create, and it's even harder to release.
- wiz21c 6y ago
  > Claiming something that is not true is in itself wrong. yup, in any project, and especially the one done for the community where the only you get is satisfaction and fame, the success is super tied to communication. Good, honest communication is what builds trust.

nullc 6y ago

Wouldn't it be nice if OpenAI were like .. actually open? :P

You-Are-Right 6y ago
Freedom is Slavery.
ryanackley 6y ago
Not just that. To even get access to their API, you need to apply. That is the future of AI I'm afraid without projects like this. That is Elites controlling AI and deciding who is "worthy" to use it. I'm sure they have the best of intentions but "worthiness" is subjective.
- TremendousJudge 6y ago
  Depends on who you mean by "they". If you mean the researchers, then sure, they probably actually believe whatever's written in their ethics statement. Now, the actual owners? I don't believe it for a second
brunoluiz 6y ago
Considering last decade social consequences due to easy access to APIs and data, I am quite happy that these initiatives are cautious around opening up software which can have huge impact on society.
- hesdeadjim 6y ago
  Unfortunately, the cat is out of the bag. Their methods are documented and the results exciting, so to a bad actor (especially state-sponsored) it's completely justified to spend millions attempting to replicate their results from what is publicly available.

deleted 6y ago

[deleted]

6gvONxR4sf7o 6y ago

People are always claiming to release replicated models by replicating the architecture (or main parts of it) but not testing whether it produces the same level of results. It's maddening, especially when the level of results is so directly measurable (just measure what the paper did, not that it's easy, just concrete).

stellaathena 6y ago
Our README has comparisons with GPT-2 and GPT-3
- 6gvONxR4sf7o 6y ago
  Is there anything on its few shot learning performance? I took few shot learning as the main point of GPT-3. Sorry if I just overlooked it, but I don’t see anything on few shot learning in the readme.