stellaathena

9 ms·

31.
by stellaathena 5y ago
That paper also claims zero-shot performance. However, the models that the bulk of the claims are made about are 20x the size of the ones considered in this paper. That's completely consistent with what I said.
32.
by stellaathena 5y ago
This paper doesn't make a whole lot of sense to me. I'm not familiar with any work that claims meaningful zero-shot performance in models as small as the ones considered in this paper. Quite the opposite, both the GPT-3 and FLAN p
33.
by stellaathena 5y ago
Minor correction: I (Stella Biderman) am a contributor to BigBench, have read many of its tasks, and have had access to it for months. However I played a rather minor role in the research, and no role in the selection of training or evaluat
34.
by stellaathena 5y ago
[Disclaimer: I am an author of the above paper and played a rather minimal role. I am also a prominent member of EleutherAI.] "Instruction-tuning" is clearly in the air. Simultaneous work at Google (released less than two weeks ag
35.
by stellaathena 5y ago
This paper is weirdly similar to one I wrote for #NeurIPS's "I Can't Believe It's Not Better!" Workshop. Had I not been constrained by the structure and rules of the workshop, I would have pretty much written this e
36.
by stellaathena 5y ago
If anyone has 300k and wants to buy a 20B parameter language model, custom trained on your data, let me know.
37.
by stellaathena 5y ago
Turning down hundreds of thousands of dollars of cloud credits because of vendor lock-in is extremely shortsighted for the overwhelming majority of projects.
38.
by stellaathena 5y ago
In Google’s defense, it’s not that Ben didn’t go to college it’s that he’s still a college student. This is less “experienced ML dev iced out over lack of degree” and more “college kid does something amazing and some people aren’t sold to h
39.
by stellaathena 5y ago
Yes, but that data is made publicly available in physics. It’s not kept as a secret so some company can make millions of dollars selling it.
40.
by stellaathena 6y ago
GPT-4 doesn’t exist mate.
41.
by stellaathena 6y ago
> Their work on GPT-Neo rules me up because they do such a weak job comparing it to the models whose hype they’re riding. Building open source infrastructure is hard. There does not currently exist a comprehensive open source framework f
42.
by stellaathena 6y ago
Hi! I’m the EAI person who your criticism of the Pile is most directed at. I’m curious if you read Sections 6.5 and 7 of the Pile working paper and, if so, what your response to it is. As you note, virtually everyone trains on copyright dat
43.
by stellaathena 6y ago
This'll probably sound silly, but we weren't really expecting much of a response. We've been sitting on these models (they were trained for the Pile paper) for months for no particular reason besides being focused on other th
44.
by stellaathena 6y ago
Our README has comparisons with GPT-2 and GPT-3
45.
by stellaathena 6y ago
We’ve added a table with some evaluation scores to the GitHub repo, and you can see a comparison between our scores, GPT-2, and GPT-3 here: https://twitter.com/BlancheMinerva/status/137399189661642752... tl;dr we
46.
by stellaathena 6y ago
We’ve added a table with some evaluation scores to the GitHub repo, and you can see a comparison between our scores, GPT-2, and GPT-3 here: https://twitter.com/BlancheMinerva/status/137399189661642752... tl;dr we
47.
by stellaathena 6y ago
I am sorry that I was misinformed about the state of our DALL-E replication when I made that tweet. It was not malicious - I was reporting what I had been told by someone else. Yes I was wrong. That said, I had hoped that maybe after two an
48.
by stellaathena 6y ago
Ada is 2.7B, Babbage is 6.7B, Curie is 13.0B, and DaVinci is 175B. The new one they announced last month is in the 20-50B range I think, not totally sure though.
49.
by stellaathena 6y ago
There's a PyTorch + DeepSpeed repository here: https://github.com/EleutherAI/gpt-neox
50.
by stellaathena 6y ago
Hi! Thanks for trying it out. There was a bug that should now be fixed. When I run the example unicorn prompt I get the follow. Don't hesitate to open an issue if you're still having trouble. "In a shocking finding, scientist
51.
by stellaathena 6y ago
It's because OpenAI sells them for profit. The "Ada" model is the same size as the larger of these two EleutherAI models.
52.
by stellaathena 6y ago
It's literally a couple people hanging out in a discord channel and doing this as a way to procrastinate their jobs.
53.
by stellaathena 6y ago
Both models were trained on Wikipedia, so that's a particularly bad choice. But yes, in practice this is what people tend to do. Take results with a very large grain of salt though, as the domain of the prompts you feed it make a huge
54.
by stellaathena 6y ago
200k emails is not enough to train a model from scratch. If you check out the google colab file in the GPT-Neo repository, it talks about how to fine-tune the model on data which is what you want to do
55.
by stellaathena 6y ago
Not hugely, but yes. I tend to think of GPT as a style of architecture with consistent themes and major features, but varying minor features and implementation details. Off the top of my head, I believe the most important difference is that
56.
by stellaathena 6y ago
That's fair. I also later learned that the title didn't explicitly mention model size at first, and I would have probably raised similar complaints had I seen that.
57.
by stellaathena 6y ago
Our current estimate is that it requires between 2000 and 4000 V100 months.
58.
by stellaathena 6y ago
GPT-3 is a model architecture, not a model. While the largest GPT-3 model is 175B, that very paper has a table that includes "GPT-3 XL" (1.3B) and "GPT-3 2.7B" as models in the GPT-3 architecture. The 2.7B model is the s
59.
by stellaathena 6y ago
The Eye has been a phenomenal partner and enables a lot of what we do. In addition to providing terabytes of storage for free, they also help us out with CPU from time to time.
60.
by stellaathena 6y ago
GPT-3 isn't a single model. It's a model architecture that is very closely followed by GPT-Neo. The 2.7B model is the exact same size as something OpenAI sells under the label "GPT-3"

More ›