leogao

6 ms·

31.
by leogao 6y ago
Handling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.
32.
by leogao 6y ago
The inference latency would also be prohibitive.
33.
by leogao 6y ago
Yeah, the Pile is approximately the size of the GPT-3 training data, which is not a coincidence--one major reason we created the Pile (though certainly not the only one) was for our GPT-3 replication project.
34.
by leogao 6y ago
Oh, there's a lot more... no promises atm, but we plan on going much bigger for our next dataset.
35.
by leogao 6y ago
EleutherAI has a very flat hierarchy; we do not have any BDFL-like figure.
36.
by leogao 6y ago
Twitter thread: https://twitter.com/nabla_theta/status/1345130412532645888
37.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling(pile.eleuther.ai)
223 points by leogao 6y ago|60 comments
38.
by leogao 6y ago
Setting aside any objections one may raise regarding the term emergence[1], the objective of this post was to discuss how while GPT-x alone cannot become an agent, its internal representations can be harnessed to create an agent. Note that
39.
by leogao 6y ago
Your doubt about more data -> better semantic knowledge about the world is well-placed and I freely admit that at this stage it's mostly conjecture, although GPT-3 does provide some evidence. However, my point is that once this hurd
40.
by leogao 6y ago
I don't doubt the input and output modules will need some work, but in the grand scheme of things probably not that much. The big missing piece imo is just models with better world models—and it doesn't look like the scaling wars
41.
by leogao 6y ago
This is a great summary!
42.
Building AGI Using Language Models(leogao.dev)
26 points by leogao 6y ago|19 comments
43.
Why GPT-3 Matters(leogao.dev)
1 points by leogao 6y ago|0 comments
44.
Why GPT-3 Matters(leogao.dev)
3 points by leogao 6y ago|0 comments
45.
GPT-3: A Brief Summary(leogao.dev)
1 points by leogao 6y ago|0 comments
46.
The Decade of Deep Learning(leogao.dev)
1 points by leogao 7y ago|0 comments
47.
The Decade of Deep Learning(leogao.dev)
2 points by leogao 7y ago|0 comments
48.
The Decade of Deep Learning(leogao.dev)
1 points by leogao 7y ago|0 comments
49.
The Difficulties of Text Generation Using Autoregressive Language Models(leogao.dev)
3 points by leogao 7y ago|0 comments