3 ms·

In my opinion They won't outperform the best LLMs for one problem: Quality of data. A hidden secret that everyone knows is that good LLMs use copyrighted data

by bratao 3y ago

In my opinion They won't outperform the best LLMs for one problem: Quality of data. A hidden secret that everyone knows is that good LLMs use copyrighted data like books-3. OpenAI itself cites a dataset from books and does not go into detail in its first papers.

giardini 3y ago
Speaking of "secrets": We didn't create an open-source project for the atomic bomb. Quite the contrary: it was one of the best-kept secrets for years. Sooooo... if LLMs are as revolutionary as some claim, shouldn't there be significant effort to keep new advances and developments secret and lead competitors astray? That is, if I knew something that would provide a 10- or 100-fold improvement in LLMs, my tendency would be to profit/benefit from that rather than publish it. And to mislead competitors? Has LLM technology development gone "underground" yet? And what is the role of the 3-letter agencies and the government in this radical development? Or is LLM technology all unicorns with rainbow farts where the entire world benefits and the creators get only thanks? Somehow I don't believe so.
giardini 3y ago
Is bigger always better? Surely "bigger is better" is at odds with "garbage in, garbage out". But I am not an LLM modeler. IOW can LLMs be improved by improving the quality and/or even *limiting the size of their "corpus"? Could we instead use, as corpus, a select body of knowledge primarily based on Western sources such as those texts a student might read as part of an excellent liberal arts education? Further, would it not be possible to limit the corpus text (and consequently later, queries) to grammatically correct language? That is, must we include all the "errors" of common speech in a corpus, such as occur especially in fiction? Indeed, could we not exclude fictional works altogether, esp. if the model is to be used for scientific work? I'd be curious to see an LLM trained on "The Great Books" plus a college course plan of textbooks. But again, I am not an LLM modeler. Anyone experimenting with such "Small Language Models(SLM)"?
- ronsor 3y ago
  The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (really high quality); however, having a lot of data seems to be able to compensate for widely varying quality. [0] https://ar5iv.labs.arxiv.org/html/2305.07759 https://ar5iv.labs.arxiv.org/html/2305.07759
  - giardini 3y ago
    Thank you for the reference! But as long as there is low quality data might not that data "pollute" the process (whatever that may be)? i.e., "GIGO". Is "low quality data" possibly necessary for a properly-working LLM?! [Aside: why the downvote?]

by bratao 3y ago

giardini 3y ago

Speaking of "secrets": We didn't create an open-source project for the atomic bomb. Quite the contrary: it was one of the best-kept secrets for years. Sooooo... if LLMs are as revolutionary as some claim, shouldn't there be significant effort to keep new advances and developments secret and lead competitors astray? That is, if I knew something that would provide a 10- or 100-fold improvement in LLMs, my tendency would be to profit/benefit from that rather than publish it. And to mislead competitors? Has LLM technology development gone "underground" yet? And what is the role of the 3-letter agencies and the government in this radical development? Or is LLM technology all unicorns with rainbow farts where the entire world benefits and the creators get only thanks? Somehow I don't believe so.

giardini 3y ago

Is bigger always better? Surely "bigger is better" is at odds with "garbage in, garbage out". But I am not an LLM modeler. IOW can LLMs be improved by improving the quality and/or even *limiting the size of their "corpus"? Could we instead use, as corpus, a select body of knowledge primarily based on Western sources such as those texts a student might read as part of an excellent liberal arts education? Further, would it not be possible to limit the corpus text (and consequently later, queries) to grammatically correct language? That is, must we include all the "errors" of common speech in a corpus, such as occur especially in fiction? Indeed, could we not exclude fictional works altogether, esp. if the model is to be used for scientific work? I'd be curious to see an LLM trained on "The Great Books" plus a college course plan of textbooks. But again, I am not an LLM modeler. Anyone experimenting with such "Small Language Models(SLM)"?

ronsor 3y ago
The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (really high quality); however, having a lot of data seems to be able to compensate for widely varying quality. [0] https://ar5iv.labs.arxiv.org/html/2305.07759 https://ar5iv.labs.arxiv.org/html/2305.07759
- giardini 3y ago
  Thank you for the reference! But as long as there is low quality data might not that data "pollute" the process (whatever that may be)? i.e., "GIGO". Is "low quality data" possibly necessary for a properly-working LLM?! [Aside: why the downvote?]