3 ms·

let's see... English speakers tend to have a vocabulary of 30k words or so depending on what exactly you measure. GPT-4 has 1.7 trillion parameters. Of cours

by rpearl 3y ago

let's see... English speakers tend to have a vocabulary of 30k words or so depending on what exactly you measure. GPT-4 has 1.7 trillion parameters. Of course scaling up is quite unpredictable in what capabilities it gets you, but It's not that much of a stretch to imagine that a GPT-4 sized model would have a reasonably sized vocabulary. Certainly worth testing if you have the resources to train such a thing!

glitchc 3y ago
It's the mapping of parameters to vocabulary claim embedded in your response that needs validation. 1.7 trillion parameters means what exactly? Let's start with working vocabulary. Working vocabulary doesn't just mean knowing n words. It means putting n words together in factorially many valid combinations to construct sentences. And 30k btw is an insane working vocabulary. Most people know 1000 words on average in English. All of their sentences are structured from those 1000 words. This is true for most cases, except certain ones like Mandarin or German, where basic words can be used to assemble more complex words. Certainly GPT-4 knows something. Presumably that something can be mapped to a working vocabulary. How large a vocabulary that is requires a testable, reproducible hypothesis supported by experimental proof. Do you have such a hypothesis with proof? Does anyone? Until we do it's just a guess.
- _a_a_a_ 3y ago
  > Most people know 1000 words on average in English Maybe I misunderstand but that sounds so stupid and wrong I don't know where to start. Standard vocab estimates are 15K - 30K, averaging about 20K (these from memory) Edit: from wiki https://en.wikipedia.org/wiki/Vocabulary#Native-language_vocabulary https://en.wikipedia.org/wiki/Vocabulary#Native-language_voc... --- As a result, estimates vary from 10,000-17,000 word families[16][19] or 17,000-42,000 dictionary words for young adult native speakers of English.[12][17] A 2016 study shows that 20-year-old English native speakers recognize on average 42,000 lemmas, ranging from 27,100 for the lowest 5% of the population to 51,700 lemmas for the highest 5%. These lemmas come from 6,100 word families in the lowest 5% of the population and 14,900 word families in the highest 5%. 60-year-olds know on average 6,000 lemmas more. [12] According to another, earlier 1995 study junior-high students would be able to recognize the meanings of about 10,000–12,000 words, whereas for college students this number grows up to about 12,000–17,000 and for elderly adults up to about 17,000 or more.[20] ---
  - Y_Y 3y ago
    Does the average include people who don't speak English? If about 4% of the world's population are native speakers and the number of words known tails off after that I can imagine it could almost be approximately true. And maybe we are counting babies in English majority countries as native speakers but they haven't learned all their 20k words yet. Of course GP's point is still invalid in that case.
  - debo_ 3y ago
    I assumed they missed a 0 and meant 10000 words.
- rpearl 3y ago
  I don't care to pay for access to gpt-4 but one could easily use one of the vocabulary estimation tests, which use some statistics plus knowledge of word appearance frequency, to estimate its vocabulary size. https://mikeinnes.io/2022/02/26/vocab https://mikeinnes.io/2022/02/26/vocab is one such test which explains the statistics ideas, and there are many others based on similar principles.
- throwaway9274 3y ago
  100% of people know the most common 1000 words. The remainder of those who know more words fall into a consistent curve across languages that follows Zipf’s law. This is different than “most people know 1000 words on average.”
- deleted 3y ago
  [deleted]
swyx 3y ago
> if you have the resources to train such a thing! estimates i've heard for training gpt4 was $500m all in for what its worth

by rpearl 3y ago

glitchc 3y ago

It's the mapping of parameters to vocabulary claim embedded in your response that needs validation. 1.7 trillion parameters means what exactly? Let's start with working vocabulary. Working vocabulary doesn't just mean knowing n words. It means putting n words together in factorially many valid combinations to construct sentences. And 30k btw is an insane working vocabulary. Most people know 1000 words on average in English. All of their sentences are structured from those 1000 words. This is true for most cases, except certain ones like Mandarin or German, where basic words can be used to assemble more complex words. Certainly GPT-4 knows something. Presumably that something can be mapped to a working vocabulary. How large a vocabulary that is requires a testable, reproducible hypothesis supported by experimental proof. Do you have such a hypothesis with proof? Does anyone? Until we do it's just a guess.

_a_a_a_ 3y ago
> Most people know 1000 words on average in English Maybe I misunderstand but that sounds so stupid and wrong I don't know where to start. Standard vocab estimates are 15K - 30K, averaging about 20K (these from memory) Edit: from wiki https://en.wikipedia.org/wiki/Vocabulary#Native-language_vocabulary https://en.wikipedia.org/wiki/Vocabulary#Native-language_voc... --- As a result, estimates vary from 10,000-17,000 word families[16][19] or 17,000-42,000 dictionary words for young adult native speakers of English.[12][17] A 2016 study shows that 20-year-old English native speakers recognize on average 42,000 lemmas, ranging from 27,100 for the lowest 5% of the population to 51,700 lemmas for the highest 5%. These lemmas come from 6,100 word families in the lowest 5% of the population and 14,900 word families in the highest 5%. 60-year-olds know on average 6,000 lemmas more. [12] According to another, earlier 1995 study junior-high students would be able to recognize the meanings of about 10,000–12,000 words, whereas for college students this number grows up to about 12,000–17,000 and for elderly adults up to about 17,000 or more.[20] ---
- Y_Y 3y ago
  Does the average include people who don't speak English? If about 4% of the world's population are native speakers and the number of words known tails off after that I can imagine it could almost be approximately true. And maybe we are counting babies in English majority countries as native speakers but they haven't learned all their 20k words yet. Of course GP's point is still invalid in that case.
- debo_ 3y ago
  I assumed they missed a 0 and meant 10000 words.
rpearl 3y ago
I don't care to pay for access to gpt-4 but one could easily use one of the vocabulary estimation tests, which use some statistics plus knowledge of word appearance frequency, to estimate its vocabulary size. https://mikeinnes.io/2022/02/26/vocab https://mikeinnes.io/2022/02/26/vocab is one such test which explains the statistics ideas, and there are many others based on similar principles.
throwaway9274 3y ago
100% of people know the most common 1000 words. The remainder of those who know more words fall into a consistent curve across languages that follows Zipf’s law. This is different than “most people know 1000 words on average.”
deleted 3y ago
[deleted]

swyx 3y ago

> if you have the resources to train such a thing! estimates i've heard for training gpt4 was $500m all in for what its worth