6 ms·

For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune.

by cm2012 4y ago

jeremyjh 4y ago
If it can produce the current results without anything like a brain - which it does - I don't see how knowing that its 100x better at pulling shit out of its ass is going to make the experience better. Yes it will become impossible to tell by talking to it that has no brain; but since we know the path that brought it there included no brains at all, it would be a mistake to think we've realized General AI. Until it actually creates some substantiative achievement, such as designing a workable cold fusion setup I'm not going to recognize it as a super-intelligence.
- yCombLinks 4y ago
  Ahh, so being as intelligent as an average intelligence is no longer sufficient to declare it intelligent. Now it must surpass all of our achievements. 99.9% of people will never "create some substantiative achievement".
  - jeremyjh 4y ago
    I said to call it super-intelligent. To demonstrate super-intelligence it would need to demonstrate real creative powers that are beyond us in both scope and direction. That isn't necessary to prove that this is productive work; but I think it is necessary to temper some of the enthusiasm I see in this thread that all but call it a super-intelligence.
    - eru 4y ago
      Maybe you are right. As an observation: a human of normal intelligence but with much better access to a calculator and to Wikipedia, or even just external storage (faster than pen-and-paper) would already be super-human.
- hanniabu 4y ago
  Considering all the info comes from human generated content, I think a better term would be Collective Intelligence rather than artificial intelligence
  - mrguyorama 4y ago
    But it doesn't have perfect representations of everything it was trained on, only probabilistic compression essentially. It's more like a bloom filter than a database.
- dougmwne 4y ago
  There go those goalposts, speeding off into the distance. Ok, so it needs to be able to invent cold fusion for you to recognize it as intelligent? Can you invent cold fusion? Have you ever invented anything at all? I would think a good measure of intelligence would be to index it against human age development milestones, not this cold fusion business.
lelanthran 4y ago
> For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune. Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single datapoint. For a human to differentiate between a cat and a dog takes, maybe, two examples of each, not a few million pictures. An adult human who sees a hotdog for the first time will have a reasonable idea of how to make their own. None of the current crop of AI do this. It's possible that we have reached a point of diminishing returns with the current path - throwing 100x resources for a 1% increase in success rates. I'd be interested in seeing approaches that don't use a neural net (or use a largely different one) and don't need millions/billions of training data text and/or images.
- fshbbdssbbgdd 4y ago
  Parent is talking about how much data the model needs for training. You are comparing that to how much data a human needs for inference. Human training data needs are quite high - several years of learning. Look up few-shot learning if you want a more fair comparison for tasks like telling apart a cat and a hot dog given a few examples.
- space_fountain 4y ago
  There are fascinating studies from people who have been blind through childhood and have their vision restored late enough that we can talk to them. For example (https://pubmed.ncbi.nlm.nih.gov/28533387/ https://pubmed.ncbi.nlm.nih.gov/28533387/). In particular it takes several months for these previously blind children to learn to distinguish faces from non faces. I recall a pop science article which I can't find the source for now that explained that people with newly acquired sight struggle to predict the border of non moving objects, though they can typically accurately predict border of moving objects and over time they learn to predict for stationary. So yes after a lifetime of video humans can quickly learn to distinquish animals they've never seen before with a few examples, but the wonder of these AIs is they seem like they're closer to that too. Certainly I can make up a way I want to some words classified, show chat gpt a few examples and it can do the classification. I think you're mistaking a generalization across a lifetime of experience with learning. And compounding this is that a newborn while not having themselves experienced anything is born with a brain that's the result of millions of years of evolution filled with lifetimes of experience. It's honestly impressive we can get the sort of performance we've gotten with only all the text on the internet and a few months
eslaught 4y ago
Hasn't it already been trained on what is effectively the entire contents of the scrapable internet? There isn't another 10x to be had there, let alone 100x. I assume that whatever future improvements we get from improving algorithms (or perhaps through throwing more compute at it), not through larger datasets.
- eru 4y ago
  There might not be another 100x of written language. But we noticed that training your neural networks on multiple tasks actually works well. So we could start feeding our models eg audio and video. With lots of webcams we can make arbitrary amounts of new video footage. That would also allow the language model to be grounded more in our 3d reality. (Granted, we only know as a general observation that training the same network for multiple tasks 'forces' that network to become better and abstract and generalise. Nobody has yet publicly demonstrated an application of that observation to training language models + video models.) Another avenue: at the moment those large language models only see each example once, if I remember right. We still have lots of techniques for augmenting training data (eg via noise and dropout etc), or even just presenting the same data multiple times without overfitting.

For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune.

by cm2012 4y ago

For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune.

jeremyjh 4y ago

If it can produce the current results without anything like a brain - which it does - I don't see how knowing that its 100x better at pulling shit out of its ass is going to make the experience better. Yes it will become impossible to tell by talking to it that has no brain; but since we know the path that brought it there included no brains at all, it would be a mistake to think we've realized General AI. Until it actually creates some substantiative achievement, such as designing a workable cold fusion setup I'm not going to recognize it as a super-intelligence.

yCombLinks 4y ago
Ahh, so being as intelligent as an average intelligence is no longer sufficient to declare it intelligent. Now it must surpass all of our achievements. 99.9% of people will never "create some substantiative achievement".
- jeremyjh 4y ago
  I said to call it super-intelligent. To demonstrate super-intelligence it would need to demonstrate real creative powers that are beyond us in both scope and direction. That isn't necessary to prove that this is productive work; but I think it is necessary to temper some of the enthusiasm I see in this thread that all but call it a super-intelligence.
  - eru 4y ago
    Maybe you are right. As an observation: a human of normal intelligence but with much better access to a calculator and to Wikipedia, or even just external storage (faster than pen-and-paper) would already be super-human.
hanniabu 4y ago
Considering all the info comes from human generated content, I think a better term would be Collective Intelligence rather than artificial intelligence
- mrguyorama 4y ago
  But it doesn't have perfect representations of everything it was trained on, only probabilistic compression essentially. It's more like a bloom filter than a database.
dougmwne 4y ago
There go those goalposts, speeding off into the distance. Ok, so it needs to be able to invent cold fusion for you to recognize it as intelligent? Can you invent cold fusion? Have you ever invented anything at all? I would think a good measure of intelligence would be to index it against human age development milestones, not this cold fusion business.

lelanthran 4y ago

> For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune. Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single datapoint. For a human to differentiate between a cat and a dog takes, maybe, two examples of each, not a few million pictures. An adult human who sees a hotdog for the first time will have a reasonable idea of how to make their own. None of the current crop of AI do this. It's possible that we have reached a point of diminishing returns with the current path - throwing 100x resources for a 1% increase in success rates. I'd be interested in seeing approaches that don't use a neural net (or use a largely different one) and don't need millions/billions of training data text and/or images.

fshbbdssbbgdd 4y ago
Parent is talking about how much data the model needs for training. You are comparing that to how much data a human needs for inference. Human training data needs are quite high - several years of learning. Look up few-shot learning if you want a more fair comparison for tasks like telling apart a cat and a hot dog given a few examples.
space_fountain 4y ago
There are fascinating studies from people who have been blind through childhood and have their vision restored late enough that we can talk to them. For example (https://pubmed.ncbi.nlm.nih.gov/28533387/ https://pubmed.ncbi.nlm.nih.gov/28533387/). In particular it takes several months for these previously blind children to learn to distinguish faces from non faces. I recall a pop science article which I can't find the source for now that explained that people with newly acquired sight struggle to predict the border of non moving objects, though they can typically accurately predict border of moving objects and over time they learn to predict for stationary. So yes after a lifetime of video humans can quickly learn to distinquish animals they've never seen before with a few examples, but the wonder of these AIs is they seem like they're closer to that too. Certainly I can make up a way I want to some words classified, show chat gpt a few examples and it can do the classification. I think you're mistaking a generalization across a lifetime of experience with learning. And compounding this is that a newborn while not having themselves experienced anything is born with a brain that's the result of millions of years of evolution filled with lifetimes of experience. It's honestly impressive we can get the sort of performance we've gotten with only all the text on the internet and a few months

eslaught 4y ago

Hasn't it already been trained on what is effectively the entire contents of the scrapable internet? There isn't another 10x to be had there, let alone 100x. I assume that whatever future improvements we get from improving algorithms (or perhaps through throwing more compute at it), not through larger datasets.

eru 4y ago
There might not be another 100x of written language. But we noticed that training your neural networks on multiple tasks actually works well. So we could start feeding our models eg audio and video. With lots of webcams we can make arbitrary amounts of new video footage. That would also allow the language model to be grounded more in our 3d reality. (Granted, we only know as a general observation that training the same network for multiple tasks 'forces' that network to become better and abstract and generalise. Nobody has yet publicly demonstrated an application of that observation to training language models + video models.) Another avenue: at the moment those large language models only see each example once, if I remember right. We still have lots of techniques for augmenting training data (eg via noise and dropout etc), or even just presenting the same data multiple times without overfitting.