4 ms·

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https:

The leaked Google memo "We have no moat, and neither does OpenAI" is instructive here: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... Original author is an ML researcher, and the crux of his argument is that most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. The remaining misses are facts that it has never seen, usually because they are so obvious that nobody thinks to write them down explicitly. And so more training data doesn't necessarily help LLM performance, unless you're either ingesting either extremely basic facts that are so obvious most adult discourse overlooks them, or you're ingesting expert knowledge that's highly specific and only discussed in a few forums. Reddit data could perhaps help with the latter, but a.) Reddit is usually not the place to go for expert discourse and b.) there are other better sources of data for it. You'd usually be better off training on trade publications, scientific journals, or fandom than Reddit. Also the LLaMa/RedPajama approach makes this stupidly easy, because you can pass around patch sets to the model trained on specific mini-corpora, and then update the weights appropriately. Hence why the author of the Google memo believes neither Google nor OpenAI has a viable moat.

bbor 3y ago
This is a great analysis that really cleared up the impact of that memo for that me. You clearly know but for any readers who aren’t clear: that memo is the opinion of one engineer at google, and far from the (apparent) opinion of the relevant execs
- refulgentis 3y ago
  It’s dead wrong, I suggest reading the original. _No one_ thinks models trained on more data would be about the same, since everything flows out of that premise…then throws Current Thing on top…it’s very unhelpful. The “patches” he refers to are LoRA and are treated as deus ex machina. They’re not, ex. playing with Stable Diffusion we can see they’re additive but they’re not nearly as good as training the original model on the data. (Disclaimer: Googler)
  - haldujai 3y ago
    > they’re not nearly as good as training the original model on the data. How are you defining "nearly as good", can you be more specific? It's obvious full fine-tuning > [PEFT tuning] but to my understanding the gap isn't that significant, as reported in various papers. (specifically with respects to language models, I'm not familiar with diffusion models).
tmpz22 3y ago
> Reddit is usually not the place to go for expert discourse Its not expert research but Reddit can be used to find probably-real-first-hand-experience on a given subject. Which is enough to prefix "reddit" to a lot of google searches. It just needs to be a marginal improvement to Google blogspam for it to have some intrinsic value.
- piyh 3y ago
  Also really depends on what kind of experts you're looking for. 99.99% of the world's experts are not writing in trade publications or scientific journals.
- hospitalJail 3y ago
  I don't even think blogspam is the problem as much as SEO-gamed content. Blogs are fine, I don't mind reading them. The classic recipe that is 95% down the page with every possible word you could google in the preceding 95% in the form of a fake story.
- enterprise_cog 3y ago
  It used to. Past 7ish years or so I saw a lot more posting that was straight up lies or people pretending to be experts that wasn’t heavily downvoted. Or maybe it was always like that and I got more aware. Either way, not something I’d want ingested into a model.
kasunbuddhi 3y ago
[flagged]
hospitalJail 3y ago
>You'd usually be better off training on trade publications, scientific journals, or fandom than Reddit. Bold claim. There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals. That is what is missing. The absurdly specific stuff that reddit gets. Sure you can answer with nonsense that sounds realistic, but you could also answer with the exact text that solved the problem in the past. Its the difference between asking the question "How do I save money at the grocery store?" and getting generic answers like "Make a budget" and getting specific answers like "Buy fresh rather than processed".
- dmonitor 3y ago
  The replies of engineers and doctors are significantly outweighed by the responses of bored teenagers.
  - hospitalJail 3y ago
    Yeah, but I imagine bored teenagers are not patrolling the embedded subreddit.
- haldujai 3y ago
  > There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals. There are also plenty of people answering nuanced questions with complete BS. It's not that bold to claim that as when taken as a whole the signal-noise is worse on Reddit than in a publication.
- marcosdumay 3y ago
  The GP ignored books for some reason. But you get very different kinds of knowledge from each of those. They are not really comparable. You won't find any useful explanation about how to solder surface electronic components in a PCB in a book, and you won't find a deep enough explanation about how to dimension an uncoupling capacitor for an amplifier in Reddit.
- michaelt 3y ago
  Well, it depends on what you're asking the LLM to generate. If I ask an LLM to write me an essay in the style of a caveman, the LLM simulates a redditor simulating a fictional caveman (who speaks english but with short words and bad grammar). After all, that's what was in the training data. If I ask the same LLM to write me an essay in the style of a Harvard professor - who's to say the same thing doesn't happen? So I can see how, when training a model, you might want it to know what's really written by a Harvard professor, and what merely professes to be.
mathisfun123 3y ago
>most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. more than anything, AI/ML/DL enables people to make claims that cannot be refuted by any application of the scientific method. it's literally "not even wrong" in exactly the same way as people claim string theory is.
- jjoonathan 3y ago
  String theory wishes it had (unique) claims as easily falsifiable as "Once you have ingested several terabytes of natural language, you know how to generate natural language." Obviously you need a page of fine print to make a claim strictly falsifiable, but complaining about the absence of fine print in casual discussion is absurdly uncharitable unless you have reason to believe that agreeable fine print couldn't be drawn up and I'm 99% sure that in this case you have so such reason.
  - mathisfun123 3y ago
    >String theory wishes are you a string theorist? are you sure that you understand the claims made by string theory and how falsifiable they are? conversely >fine print couldn't be drawn up and I'm 99% sure... where does your 99% surety derive from? As someone trained both in mathematical physics and DL, i'm here to burst your bubble: the falsifiability of both disciplines is exacty the same because they're both premised on knowledged derived of models with an astronomical number of free parameters. just to be clear: i'm not poopooing LLN here, i.e. the models themselves are fine, i'm talking about absurd meta-claims (like ops about generating language) derived of observing many such models.
samrus 3y ago
One thing to consider is that while the large transformers used in LLMs might have these diminishing returns, we don't know what discrete jump in model architecture might come next. That model might gain a lot from even more training data. And might gain more from the semi structured data on reddit than the slightly less structured data on Wikipedia and Twitter Maybe
- danudey 3y ago
  I would argue that Wikipedia is much more structured than Reddit. > we don't know what discrete jump in model architecture might come next This is true, but as long as these models are based on creating sentences based on what sentences that it's seen have looked like, and not on fetching and understanding verifiable facts, these services will do more harm than good overall. If anything, these models need two sources of training data: 1. The standard language model (as it is now) to be able to generate and process queries and provide understandable answers 2. A database of verifiable factual information that it can query in order to prevent it from completely hallucinating information and then asserting that it is verifiable factual information when asked [0]. Until we can solve the AI hallucination problem, these systems are going to require users to be much more careful with information they're given than most people can manage right now. [0] https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avianca/ https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avi...
cbkeller 3y ago
The author links this in the fourth paragraph; their argument is that while compute is not a viable moat, training data may be.
dmazin 3y ago
> you're ingesting expert knowledge that's highly specific and only discussed in a few forums. Reddit data could perhaps help with the latter This is exactly where I think Reddit's value lies. I disagree, though, that people don't go to Reddit for it. Here are a couple recent queries where I appended reddit to my search query: * best places to visit from London * best mattress sold in UK * should I bring king mattress from US to UK (Can you tell I just moved to London?) Why do people go to Reddit? Because it's guaranteed (at least for now) that the answer came from humans who are not trying to sell you something. They may be wrong but... it's opinions.
- dingnuts 3y ago
  > it's guaranteed (at least for now) that the answer came from humans who are not trying to sell you something. No no no no! This is not guaranteed AT ALL on Reddit! Why do you think Redditors are constantly accusing each other of being shills? Because Reddit users are often corporate representatives doing "native marketing" or whatever they call it, and they are not upfront about it, because they are preying on the naivete of people like you! Seeing the belief expressed that Redditors aren't trying to sell you something is.. distressing to me. Half of Reddit is some kind of ad, if not for a physical product it's for a political party or cause, or some celebrity's personal brand. Please have a little cynicism online.. please, for the sake of everyone..
aleem 3y ago
The only thing that didn't sit well with a lot of people about the leaked memo is that it ignored the quality of GPT4 vs GPT3 and made claims that all LLMs were poised to be on par, yet that isn't true till now. What it also ignored (along with some of the comments here) is data ranking. Google didn't just build a search engine by crawling more of the web -- many search engines before it had already done that. Google managed to rank what's relevant and what isn't. Relevancy is hard. Similarly, not all scientific publications are ranked equally. Or for that matter, even publications with a lot of peer reviews or citations can become obsolete through new discoveries. Reddit's data has value in that it can fill in a lot of the gaps left by more qualitative sources and furthermore the data is user-ranked by a trusted community. This also has implications for specialised querying, for example training on just r/fitness could be fairly useful for that community. As a side note, other valuable data stores are not just text but voice/video as well. YouTube and podcast transcripts are readily available, for example to Google. Data and ranking is valuable all over again.
whomst 3y ago
the end game is MechanicGPT trained on the obscure phpBB/vBulletin car mechanic boards from the early 00's

bbor 3y ago

This is a great analysis that really cleared up the impact of that memo for that me. You clearly know but for any readers who aren’t clear: that memo is the opinion of one engineer at google, and far from the (apparent) opinion of the relevant execs

refulgentis 3y ago
It’s dead wrong, I suggest reading the original. _No one_ thinks models trained on more data would be about the same, since everything flows out of that premise…then throws Current Thing on top…it’s very unhelpful. The “patches” he refers to are LoRA and are treated as deus ex machina. They’re not, ex. playing with Stable Diffusion we can see they’re additive but they’re not nearly as good as training the original model on the data. (Disclaimer: Googler)
- haldujai 3y ago
  > they’re not nearly as good as training the original model on the data. How are you defining "nearly as good", can you be more specific? It's obvious full fine-tuning > [PEFT tuning] but to my understanding the gap isn't that significant, as reported in various papers. (specifically with respects to language models, I'm not familiar with diffusion models).

tmpz22 3y ago

> Reddit is usually not the place to go for expert discourse Its not expert research but Reddit can be used to find probably-real-first-hand-experience on a given subject. Which is enough to prefix "reddit" to a lot of google searches. It just needs to be a marginal improvement to Google blogspam for it to have some intrinsic value.

piyh 3y ago
Also really depends on what kind of experts you're looking for. 99.99% of the world's experts are not writing in trade publications or scientific journals.
hospitalJail 3y ago
I don't even think blogspam is the problem as much as SEO-gamed content. Blogs are fine, I don't mind reading them. The classic recipe that is 95% down the page with every possible word you could google in the preceding 95% in the form of a fake story.
enterprise_cog 3y ago
It used to. Past 7ish years or so I saw a lot more posting that was straight up lies or people pretending to be experts that wasn’t heavily downvoted. Or maybe it was always like that and I got more aware. Either way, not something I’d want ingested into a model.

kasunbuddhi 3y ago

[flagged]

hospitalJail 3y ago

>You'd usually be better off training on trade publications, scientific journals, or fandom than Reddit. Bold claim. There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals. That is what is missing. The absurdly specific stuff that reddit gets. Sure you can answer with nonsense that sounds realistic, but you could also answer with the exact text that solved the problem in the past. Its the difference between asking the question "How do I save money at the grocery store?" and getting generic answers like "Make a budget" and getting specific answers like "Buy fresh rather than processed".

dmonitor 3y ago
The replies of engineers and doctors are significantly outweighed by the responses of bored teenagers.
- hospitalJail 3y ago
  Yeah, but I imagine bored teenagers are not patrolling the embedded subreddit.
haldujai 3y ago
> There are plenty of engineers and doctors on reddit answering nuanced questions that arent talked about in any journals. There are also plenty of people answering nuanced questions with complete BS. It's not that bold to claim that as when taken as a whole the signal-noise is worse on Reddit than in a publication.
marcosdumay 3y ago
The GP ignored books for some reason. But you get very different kinds of knowledge from each of those. They are not really comparable. You won't find any useful explanation about how to solder surface electronic components in a PCB in a book, and you won't find a deep enough explanation about how to dimension an uncoupling capacitor for an amplifier in Reddit.
michaelt 3y ago
Well, it depends on what you're asking the LLM to generate. If I ask an LLM to write me an essay in the style of a caveman, the LLM simulates a redditor simulating a fictional caveman (who speaks english but with short words and bad grammar). After all, that's what was in the training data. If I ask the same LLM to write me an essay in the style of a Harvard professor - who's to say the same thing doesn't happen? So I can see how, when training a model, you might want it to know what's really written by a Harvard professor, and what merely professes to be.

mathisfun123 3y ago

>most weights in a LLM are significantly overdetermined. Once you have ingested several terabytes of natural language, you know how to generate natural language. more than anything, AI/ML/DL enables people to make claims that cannot be refuted by any application of the scientific method. it's literally "not even wrong" in exactly the same way as people claim string theory is.

jjoonathan 3y ago
String theory wishes it had (unique) claims as easily falsifiable as "Once you have ingested several terabytes of natural language, you know how to generate natural language." Obviously you need a page of fine print to make a claim strictly falsifiable, but complaining about the absence of fine print in casual discussion is absurdly uncharitable unless you have reason to believe that agreeable fine print couldn't be drawn up and I'm 99% sure that in this case you have so such reason.
- mathisfun123 3y ago
  >String theory wishes are you a string theorist? are you sure that you understand the claims made by string theory and how falsifiable they are? conversely >fine print couldn't be drawn up and I'm 99% sure... where does your 99% surety derive from? As someone trained both in mathematical physics and DL, i'm here to burst your bubble: the falsifiability of both disciplines is exacty the same because they're both premised on knowledged derived of models with an astronomical number of free parameters. just to be clear: i'm not poopooing LLN here, i.e. the models themselves are fine, i'm talking about absurd meta-claims (like ops about generating language) derived of observing many such models.

samrus 3y ago

One thing to consider is that while the large transformers used in LLMs might have these diminishing returns, we don't know what discrete jump in model architecture might come next. That model might gain a lot from even more training data. And might gain more from the semi structured data on reddit than the slightly less structured data on Wikipedia and Twitter Maybe

danudey 3y ago
I would argue that Wikipedia is much more structured than Reddit. > we don't know what discrete jump in model architecture might come next This is true, but as long as these models are based on creating sentences based on what sentences that it's seen have looked like, and not on fetching and understanding verifiable facts, these services will do more harm than good overall. If anything, these models need two sources of training data: 1. The standard language model (as it is now) to be able to generate and process queries and provide understandable answers 2. A database of verifiable factual information that it can query in order to prevent it from completely hallucinating information and then asserting that it is verifiable factual information when asked [0]. Until we can solve the AI hallucination problem, these systems are going to require users to be much more careful with information they're given than most people can manage right now. [0] https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avianca/ https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avi...

cbkeller 3y ago

The author links this in the fourth paragraph; their argument is that while compute is not a viable moat, training data may be.

dmazin 3y ago

> you're ingesting expert knowledge that's highly specific and only discussed in a few forums. Reddit data could perhaps help with the latter This is exactly where I think Reddit's value lies. I disagree, though, that people don't go to Reddit for it. Here are a couple recent queries where I appended reddit to my search query: * best places to visit from London * best mattress sold in UK * should I bring king mattress from US to UK (Can you tell I just moved to London?) Why do people go to Reddit? Because it's guaranteed (at least for now) that the answer came from humans who are not trying to sell you something. They may be wrong but... it's opinions.

dingnuts 3y ago
> it's guaranteed (at least for now) that the answer came from humans who are not trying to sell you something. No no no no! This is not guaranteed AT ALL on Reddit! Why do you think Redditors are constantly accusing each other of being shills? Because Reddit users are often corporate representatives doing "native marketing" or whatever they call it, and they are not upfront about it, because they are preying on the naivete of people like you! Seeing the belief expressed that Redditors aren't trying to sell you something is.. distressing to me. Half of Reddit is some kind of ad, if not for a physical product it's for a political party or cause, or some celebrity's personal brand. Please have a little cynicism online.. please, for the sake of everyone..

aleem 3y ago

The only thing that didn't sit well with a lot of people about the leaked memo is that it ignored the quality of GPT4 vs GPT3 and made claims that all LLMs were poised to be on par, yet that isn't true till now. What it also ignored (along with some of the comments here) is data ranking. Google didn't just build a search engine by crawling more of the web -- many search engines before it had already done that. Google managed to rank what's relevant and what isn't. Relevancy is hard. Similarly, not all scientific publications are ranked equally. Or for that matter, even publications with a lot of peer reviews or citations can become obsolete through new discoveries. Reddit's data has value in that it can fill in a lot of the gaps left by more qualitative sources and furthermore the data is user-ranked by a trusted community. This also has implications for specialised querying, for example training on just r/fitness could be fairly useful for that community. As a side note, other valuable data stores are not just text but voice/video as well. YouTube and podcast transcripts are readily available, for example to Google. Data and ranking is valuable all over again.

whomst 3y ago

the end game is MechanicGPT trained on the obscure phpBB/vBulletin car mechanic boards from the early 00's