3 ms·

While I have my own reasons to disagree with the worst-case scenarios of AI alarmists ("radical and irreconcilable differences in presuppositions about the most

by jerf 3y ago

While I have my own reasons to disagree with the worst-case scenarios of AI alarmists ("radical and irreconcilable differences in presuppositions about the most fundamental aspects of reality"), an awful lot of arguing about why they are wrong really boils down to just argument from incredulity. Argument from incredulity is actually not a terrible argument in general, in my opinion. It's nominally a fallacy but that just means it's not valid from an Aristotelian perspective in that it can 100% prove a statement from a previous statement, but a lot of Aristotelian fallacies are still useful in the real world when used more carefully intelligently. However, the exact point where argument from incredulity is weakest is long-term projections of how the future might be different, and that's exactly what we're talking about here. It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. It seems likely to me that Large Language Models will all have this characteristic, just by their nature. It is not the sort of thing they do. Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set, to be very very sloppy with my terminology but good enough for now. But it is very far from clear to me that this is true of all possible AIs we may produce, even in the near future. I don't know what the next step will be, I'm just confident there will be one.

famouswaffles 3y ago
>Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong. LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.
- jerf 3y ago
  Which is why I disclaimed it. I hate it when people quote things, cut off the quote, then bitch about the part they cut off. No the models will not "predict Einstein". They'll predict the most popular interpretation of him at best, and while they is also a simplification, ChatGPT is not sitting on top of the solution to the Grand Unified Theory. It may give a good overview of the consensus, but it will not be able to tell you the correct solution to the problem right now... though it won't be hard to convince it to swear up and down that it has.
  - jameshart 3y ago
    For ‘predict Einstein’ read ‘predict what an Einstein-level intellect would think (about a given subject)’, not literally replicate what Einstein did. You’re focused on the idea of LLMs as collators of ‘things that have been said’, but that’s not all that they collate from their training set.
    - legutierr 3y ago
      >> but that’s not all that they collate from their training set OK, so what else is it?
      - bigyikes 3y ago
        A generalized model of reality that has increased in fidelity as LLMs have improved.
      - famouswaffles 3y ago
        LLMs learn to make predictions. They don't learn to imitate. They don't learn to simulate. There's nothing about learning to predict that makes the intelligence you gain constrained to the data you're learning from. The opposite if anything. But that's another argument for another time. The point I was making is that LLMs don't come out of training the average of what they've learned. They can make predictions on any state in their training. They can make predictions about the most intelligent state and the dumbest state. The most emotional state and the least emotional state. It's this powerful prediction range that makes them capable of imitating or simulating damn near anything in the training set to high and ever increasing accuracy.
    - jerf 3y ago
      Yes, I'm aware of that too. Maybe I just don't feel like spewing a complete description of LLMs into my every post about them. To the extent that they may come up with novel ideas, they have no ability to compare them against the true state of the world. This is not exactly a limitation of them per se that could be overcome with more computation, so much as just a structural fact about them; they have no loop where they can form a hypothesis, test it, and adjust based on data. It simply doesn't exist. Which is part of why I keep saying that while I'm less impressed than everyone else is with LLMs, the future AIs that will incorporate them but not simply be an LLM is going to really knock people's socks off. Pretty much all the things people trying to convince LLMs to do that they really can't do are going to work in that generation. I have no idea if that generation is six months or six years away but I wouldn't bet much more than a few years.
visarga 3y ago
> It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. I think it could do that in software. Assuming compute is no issue we have: - LLMs writing code, explaining code, changing code, and observing code execution - LLMs that understand ML concepts and can explain their own workings - LLMs can generate the training set all from inside (see TinyStories) - LLMs can make "RLHF" data for the fine-tuning (see Alpaca, tuned with GPT3.5 and GPT4 data from LLaMA) If we take a look, it seems LLMs can self replicate in software with nothing else but compute and a neural net framework. Of course making the chips is a whole other story.
mike_hearn 3y ago
I don't know what the next step will be, I'm just confident there will be one. But we don't know when. AI is a bit unusual in that it had a winter, unlike most other aspects of computing which have seen much more consistent progress. Given past performance it's entirely possible that progress in AI will just stall. Arguably it had already stalled thanks to there being only a handful of companies that were able and still interested/funded to create models, and most of those decided not to actually let anyone use the results. OpenAI dominates mindshare exactly because there are so few organizations that both can and will do this stuff well. So there's lots of ways AI progress could go off the rails again.

by jerf 3y ago

famouswaffles 3y ago

>Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong. LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.

jerf 3y ago
Which is why I disclaimed it. I hate it when people quote things, cut off the quote, then bitch about the part they cut off. No the models will not "predict Einstein". They'll predict the most popular interpretation of him at best, and while they is also a simplification, ChatGPT is not sitting on top of the solution to the Grand Unified Theory. It may give a good overview of the consensus, but it will not be able to tell you the correct solution to the problem right now... though it won't be hard to convince it to swear up and down that it has.
- jameshart 3y ago
  For ‘predict Einstein’ read ‘predict what an Einstein-level intellect would think (about a given subject)’, not literally replicate what Einstein did. You’re focused on the idea of LLMs as collators of ‘things that have been said’, but that’s not all that they collate from their training set.
  - legutierr 3y ago
    >> but that’s not all that they collate from their training set OK, so what else is it?
    - bigyikes 3y ago
      A generalized model of reality that has increased in fidelity as LLMs have improved.
    - famouswaffles 3y ago
      LLMs learn to make predictions. They don't learn to imitate. They don't learn to simulate. There's nothing about learning to predict that makes the intelligence you gain constrained to the data you're learning from. The opposite if anything. But that's another argument for another time. The point I was making is that LLMs don't come out of training the average of what they've learned. They can make predictions on any state in their training. They can make predictions about the most intelligent state and the dumbest state. The most emotional state and the least emotional state. It's this powerful prediction range that makes them capable of imitating or simulating damn near anything in the training set to high and ever increasing accuracy.
  - jerf 3y ago
    Yes, I'm aware of that too. Maybe I just don't feel like spewing a complete description of LLMs into my every post about them. To the extent that they may come up with novel ideas, they have no ability to compare them against the true state of the world. This is not exactly a limitation of them per se that could be overcome with more computation, so much as just a structural fact about them; they have no loop where they can form a hypothesis, test it, and adjust based on data. It simply doesn't exist. Which is part of why I keep saying that while I'm less impressed than everyone else is with LLMs, the future AIs that will incorporate them but not simply be an LLM is going to really knock people's socks off. Pretty much all the things people trying to convince LLMs to do that they really can't do are going to work in that generation. I have no idea if that generation is six months or six years away but I wouldn't bet much more than a few years.

visarga 3y ago

> It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. I think it could do that in software. Assuming compute is no issue we have: - LLMs writing code, explaining code, changing code, and observing code execution - LLMs that understand ML concepts and can explain their own workings - LLMs can generate the training set all from inside (see TinyStories) - LLMs can make "RLHF" data for the fine-tuning (see Alpaca, tuned with GPT3.5 and GPT4 data from LLaMA) If we take a look, it seems LLMs can self replicate in software with nothing else but compute and a neural net framework. Of course making the chips is a whole other story.

mike_hearn 3y ago

I don't know what the next step will be, I'm just confident there will be one. But we don't know when. AI is a bit unusual in that it had a winter, unlike most other aspects of computing which have seen much more consistent progress. Given past performance it's entirely possible that progress in AI will just stall. Arguably it had already stalled thanks to there being only a handful of companies that were able and still interested/funded to create models, and most of those decided not to actually let anyone use the results. OpenAI dominates mindshare exactly because there are so few organizations that both can and will do this stuff well. So there's lots of ways AI progress could go off the rails again.