5 ms·

> They'll run out of data, and the models won't be able to get any better. Well, that is enough to prove to me he doesn't know enough about the technology for

> They'll run out of data, and the models won't be able to get any better. Well, that is enough to prove to me he doesn't know enough about the technology for me to pay attention to his opinions...

alangibson 2y ago
On this point you're probably right. Post your counterargument. This is HN after all.
- BobbyJo 2y ago
  Counter argument: The data necessary to train these models has been around for a while, but didnt lead to a breakthrough. What lead to breakthrough performance was new model architectures, specifically attention mechanisms. There is nothing that says you can't make much better models using the exact same data by changing the architecture of the network. In fact, that's exactly what we've been doing for the last 15 years.
gala8y 2y ago
How much of existing data did they devour? - internet (incl. blogosphere, all www, forums, social networks, dark net,...) - all of digitized books - all statistical data, governments data, military data - all scientific papers and books - all commercial companies data - all private data/personal data - ADDED - all video/audio data - all new data... If someone has some links on what the current state of affairs is (what they really trained on), I would be really interested.
- wongarsu 2y ago
  Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it. I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means. Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes). Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget
  - gala8y 2y ago
    > Training data is the competitive edge, so everyone refuses to elaborate about what they use. I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.
  - dns_snek 2y ago
    Regarding their ethical concerns or lack thereof, OpenAI Whisper will often hallucinate transcriptions such as "Subtitles by <random nickname>" which strongly indicates that it's trained on ripped movies and crowdsourced subtitles found online.
xracy 2y ago
"What can be asserted without evidence can also be dismissed without evidence" By this razor, I feel like I can dismiss your opinion. Can you elaborate why you think he doesn't know enough about technology for you to pay attention to his opinions? Is it cause you have proof that this is false? Cause it's predicting the future, there are an infinite number of ways it can go.

> They'll run out of data, and the models won't be able to get any better. Well, that is enough to prove to me he doesn't know enough about the technology for me to pay attention to his opinions...

alangibson 2y ago

On this point you're probably right. Post your counterargument. This is HN after all.

BobbyJo 2y ago
Counter argument: The data necessary to train these models has been around for a while, but didnt lead to a breakthrough. What lead to breakthrough performance was new model architectures, specifically attention mechanisms. There is nothing that says you can't make much better models using the exact same data by changing the architecture of the network. In fact, that's exactly what we've been doing for the last 15 years.

gala8y 2y ago

How much of existing data did they devour? - internet (incl. blogosphere, all www, forums, social networks, dark net,...) - all of digitized books - all statistical data, governments data, military data - all scientific papers and books - all commercial companies data - all private data/personal data - ADDED - all video/audio data - all new data... If someone has some links on what the current state of affairs is (what they really trained on), I would be really interested.

wongarsu 2y ago
Training data is the competitive edge, so everyone refuses to elaborate about what they use. Also they probably don't want to invite anyone to sue them over it. I think there's some quote that GPT-4 is trained on a significant fraction of the entire internet, whatever that means. Beyond that it's hard to tell. If I were in charge of curating these data sets and not afraid of lawsuits or ethical concerns I would certainly add the entire contents of library genesis, anna's archive, sci-hub, etc. I'd also get a deal with the guy who has scraped 5 billion discord messages (he even explicitly offers the whole dataset for LLM training purposes). Those are the low-hanging fruit. There is still a lot of knowledge that isn't on the internet or that isn't accessible without an account. But that's mostly a question of your budget
- gala8y 2y ago
  > Training data is the competitive edge, so everyone refuses to elaborate about what they use. I absolutely understand it, thus curiosity about some leaked insiders knowledge or anything. Anyway, given availability of these huge data sets (incl. ones you mention) and advancements in processing techniques, we probably have a long way to go. Also, video looks like two guys talking about everything except AI bubble.
- dns_snek 2y ago
  Regarding their ethical concerns or lack thereof, OpenAI Whisper will often hallucinate transcriptions such as "Subtitles by <random nickname>" which strongly indicates that it's trained on ripped movies and crowdsourced subtitles found online.

xracy 2y ago

"What can be asserted without evidence can also be dismissed without evidence" By this razor, I feel like I can dismiss your opinion. Can you elaborate why you think he doesn't know enough about technology for you to pay attention to his opinions? Is it cause you have proof that this is false? Cause it's predicting the future, there are an infinite number of ways it can go.