3 ms·

> Already you can see this: whatever language-learning system we’re all meant to worship at the feet of ... This is a ridiculous strawman. > has now been trai

by rytill 5y ago

> Already you can see this: whatever language-learning system we’re all meant to worship at the feet of ... This is a ridiculous strawman. > has now been trained on all the natural language that exists on the internet, in order to produce results which are not, in fact, acceptable. And there’s nowhere to go from here: there is no more training data. Better algorithms and hardware? Also, no one has ever trained anything on "all the natural language which exists on the internet". Only a small (curated) fraction of that is captured in datasets. 45TB of text data was used to train GPT-3.

jcranmer 5y ago
> 45TB of text data was used to train GPT-3. A kilobyte of text is roughly half a page of text. If the 45TB does not include the size of metadata, then that comes out to roughly 3,000 pages of text per living person on the planet. Especially when you consider how little of the world's population is online, and how much writing isn't preserved, that would probably be a double-digit percentage of the total extant digital written record of humans. (This makes me think that the size does include metadata size or markup. But even if it includes those things, you're still probably looking at 0.1-10% of the extant digital written record. The only way it's not a significant fraction of all extant human writing is if the size is also including copious amounts of non-text data, such as embedded images or videos.)
- sengstrom 5y ago
  Let's have google check that: https://www.google.com/search?q=45+TB+%2F+8+billion https://www.google.com/search?q=45+TB+%2F+8+billion That comes out to about 3 pages by your measure.
  - jcranmer 5y ago
    Dang, you're right; I calculated 1 TB = 1 trillion bytes, when I should have calculated it as 1 billion kilobytes. (So much for decimal-based systems making it easy to do calculations in your head...)
gwern 5y ago
You're also incorrect, although less wrong than OP. 45TB was the starting point (the Common Crawl snapshots, mostly). The actual amount of plain text fed in is much smaller, they filtered it down to like 500GBish and then didn't even train a full epoch. (The data scaling law is also favorable, so the dataset doesn't need to grow much even for much larger smarter models.)

by rytill 5y ago

jcranmer 5y ago

> 45TB of text data was used to train GPT-3. A kilobyte of text is roughly half a page of text. If the 45TB does not include the size of metadata, then that comes out to roughly 3,000 pages of text per living person on the planet. Especially when you consider how little of the world's population is online, and how much writing isn't preserved, that would probably be a double-digit percentage of the total extant digital written record of humans. (This makes me think that the size does include metadata size or markup. But even if it includes those things, you're still probably looking at 0.1-10% of the extant digital written record. The only way it's not a significant fraction of all extant human writing is if the size is also including copious amounts of non-text data, such as embedded images or videos.)

sengstrom 5y ago
Let's have google check that: https://www.google.com/search?q=45+TB+%2F+8+billion https://www.google.com/search?q=45+TB+%2F+8+billion That comes out to about 3 pages by your measure.
- jcranmer 5y ago
  Dang, you're right; I calculated 1 TB = 1 trillion bytes, when I should have calculated it as 1 billion kilobytes. (So much for decimal-based systems making it easy to do calculations in your head...)

gwern 5y ago

You're also incorrect, although less wrong than OP. 45TB was the starting point (the Common Crawl snapshots, mostly). The actual amount of plain text fed in is much smaller, they filtered it down to like 500GBish and then didn't even train a full epoch. (The data scaling law is also favorable, so the dataset doesn't need to grow much even for much larger smarter models.)