15 ms·

I've been wondering about something: (I only know the basics of AI, this might be kinda incoherent) Right now if you look at GTP-3's output it seems like it's

I've been wondering about something: (I only know the basics of AI, this might be kinda incoherent) Right now if you look at GTP-3's output it seems like it's approaching a convincing approximation of a bluffing college student writing a bad paper, correct sentences and stuff but very 'cocky'. It cannot tell right from wrong, and it will just make up convincing rubbish, 'hoping' to fool the reader. (I know I'm anthropomorphizing but bear with me). Current models are being trained on a huge amount of internet text. As smarmy denizens of hackernews we know that people are very often wrong (or 'not even wrong') on the internet. It seems to me that anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? We've seen with a lot of machine-learning stuff that biased data will create biased models, so you have to be really careful what you train it on. The dataset on which GTP-n has to be trained has to be pretty huge(?); and moderation is hard(?) and doesn't scale; it's easier to generate falsehood than truth; and the further we go along the more of internet data will be (weaponized?) output of GTP-(n-1); So won't the arrival of AGI just be sabotaged by the arrival of AGI? Has anyone written something about the process of building AGI that deals with this?

tda 6y ago
> It would be a bit like how carbon dating or production of low background steel changed after 1945 due to nuclear testing https://news.ycombinator.com/item?id=23896293 https://news.ycombinator.com/item?id=23896293 I guess books published will be more useful than reddit rants (depending on the application)
deleted 6y ago
[deleted]
lordnacho 6y ago
I've been wondering about something similar to you, but I read one of Pearl's causality books recently and thought that might be the missing piece. It's certainly impressive what GPT-3 can do, but it boggles the mind how much data went into it. By contrast a well-educated renaissance man might have read a book every month or so from age 15 to 30? That doesn't seem to be anywhere near what GPT could swallow in a few seconds. When you look at how GPT answers things, it kinda feels like someone who has heard the keywords and can spout some things that at least obscure whether it has ever studied a given subject, and this is impressive. What I wonder is whether it can do reasoning of the causality kind: what if X hadn't happened, what evidence do we need to collect to know if theory Z is falsified, which data W is confounding? To me it seems that sort of thing is what smart people are able to work out, with a lot of reading, but not quite the mountain that GPT reads.
- skulk 6y ago
  > By contrast a well-educated renaissance man might have read a book every month or so from age 15 to 30? That doesn't seem to be anywhere near what GPT could swallow in a few seconds. You're ignoring the insane amount of sensory information a human gets in 30 years. I think that absolutely dwarfs the amount of information that GPT-3 eats in a training run.
  - lordnacho 6y ago
    That's an interesting point. I'm not sure how to measure that though. Also my guess is we have the sensors on for part of the day only, plus that's filtered heavily by your attention process, eg you can't read two books at once.
  - PaulDavisThe1st 6y ago
    But that sensory information includes very few written words. GPT(n) isn't being trained on "worldly audio data", or "worldly tactile data", or in fact any sensory data at all. So the two training sets are completely orthogonal, and the well educated renaissance man is somehow able to take a very small exposure to written words and do at least as well as GPT(n) in processing them and responding.
    - klipt 6y ago
      Did this renaissance man teach themselves to read from scratch or are we assuming they were assisted in their schooling?
      - PaulDavisThe1st 6y ago
        Doesn't make too much difference to the overall point.
    - MauranKilom 6y ago
      And the renaissance man has tons of structure encoded in his brain on birth already. Just like GPT-3 does before you give it a prompt. I'm not saying this is fully equivalent (clearly a baby can't spout correct Latex just by seeing three samples), but you simply cannot just handwave away thousands of years of human evolution and millions of years of general evolution before that. The renaissance man is very obviously not working solely based on a few years of reading books (or learning to speak/write).
Veedrac 6y ago
> It cannot tell right from wrong It doesn't know which facts about the world are true and which are fabricated except through the text it's trained on, but to a large extent neither do you. It suffices to merely have the ability to reason about it. The primary difference is that whereas you reason fairly competently from one perspective with one pseudo-coherent set of goals, GPT-3 reasons to a weak degree from all perspectives, privileging no view point in particular. How important this ends up being depends on where the model plateaus. On one extreme, if it plateaus close to where GPT-3 already is, no harm done, it's a fun toy. On the other, if it scales until perplexity gets to far-superhuman levels, it doesn't matter at all, since you can just prompt it with Terence Tao talking about his latest discovery. Naturally, it will land somewhere in the middle of these points. The question is ultimately then whether the landing point captures enough general reasoning that you can use it to bootstrap some more advanced reasoning agent. A sufficiently powerful GPT-N should, for example, be able to deliberate over its own generated ideas and sort the coherent reasoning from the incoherent.
rytill 6y ago
I've also been wondering this. It is like Kessler Syndrome. We have to be careful not to pollute our ecosystem of data. https://en.wikipedia.org/wiki/Kessler_syndrome https://en.wikipedia.org/wiki/Kessler_syndrome Speaking with GPT-3 also makes one realize how influenced its predictions of AI scenarios are by dystopian memes. In any conversation in which you are "speaking to the AI", the AI can go rogue. There just aren't enough positive role models authors have written for AGI.
deleted 6y ago
[deleted]
simonh 6y ago
I don't think the problem is really with the data set, it's that GTP-3 doesn't actually have any understanding of the data. It's building a model of the text input to generate text output. It's not building a model of what the data means, or what it represents. When GTP-3 writes a scientific paper it's not trying to test a premise or critically evaluate some data, it's trying to generate text that looks like that sort of thing. It doesn't matter how many excellent quality papers you fed it, or how stringently you excluded low quality input data, it would still only be trying to produce facsimiles. It wouldn't be actually trying to do the things a real conscientious scientist is trying to do when they write a paper. Arguably at best it might be trying to do what a deceitful scientist trying to get credit for a paper with spurious results with no actual scientific merit might be doing when writing a paper that looks plausible, but thats actually a completely different activity. The code generation examples are really interesting. Here it's being used to generate real working code that has actual value, and it appears to work pretty well. It's code often has bugs, but who's doesn't? It only works for fairly short precisely definable coding tasks though. I don't think scaling it up to more complex coding problems is going to work. Again it doesn't understand the meaning of anything. It's trying to produce code that looks like working code, not actually solve the programming task you're giving it. It doesn't even know what a program is or what a programming task is. It doesn't know what input and output are. For example you can't ask it to modify existing code to change it's behaviour. It doesn't know code has behaviour. It has no idea what that even means and has no way to find out or any route to gaining that capability, because that's not a text transformation task and all it does is transform text.
- SpicyLemonZest 6y ago
  How can we tell whether or not GPT-3 has understanding of the data?
  - deleted 6y ago
    [deleted]
  - grugagag 6y ago
    I think the best way to tell whether GPT-3 has understanding of its data is by by asking questions related to the data but not explicit in the training dataset.
giancarlostoro 6y ago
> It seems to me that anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? Ah one only needs to think back to Tay to know how these sort of things will end. https://en.wikipedia.org/wiki/Tay_(bot) https://en.wikipedia.org/wiki/Tay_(bot) (Imagine 4chan got a wind of this bot and retrained it, which is probably what happened...)
Smaug123 6y ago
> it will just make up convincing rubbish It's not certain that this is always the case. In at least one case I've seen, if you give it question-and-answer prompts where you don't demonstrate that you will accept the answer "your question is nonsense", it will indeed make things up; but if you include "your question is nonsense" as an acceptable answer in the sample prompts, then it will use it correctly. See https://twitter.com/nicklovescode/status/1284050958977130497 https://twitter.com/nicklovescode/status/1284050958977130497 . It seems that we have a lot to learn about how to use GPT-3 effectively!
deleted 6y ago
[deleted]
resu_nimda 6y ago
It seems to me that anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? That same sentiment could be equally applied to humans, and not just in the internet era, but throughout all of history. There will always be misinformation and "wrong" opinions out there. "It cannot tell right from wrong" is an accusation leveled against human beings every day. We can't even all agree on what is right and wrong, truth or untruth. A true AI is going to have to wade through all that and make its own decisions to be viewed and judged from many different perspectives, just like the rest of us.
confeit 6y ago
> anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? Low-quality noise cancels out and leaves the high-quality signal. In the limit, the internet offers the true sequence probabilities for compression of natural text. You can also put more weight on authoritative data sources, such as Wikipedia and StackOverflow, but even uniformly weighted: It is possible to sequence-complete prime numbers, despite the many many pages online with random numbers. GPT-3 is trained on a filtered version of Common Crawl, enhanced with authoritative datasets, such as Books1, WebText, and Wikipedia-en. Moderation is done automatically, with a toxicity classifier/toggle. If GPT-n becomes good enough to be accepted in authoritative datasets, then it is perfectly fine training data, a form of semi-supervised learning. Bias is going to be a double-edged sword: I believe it will be impossible to prescribe common sense, nor to sanitize common sense to remove, say, gender bias, and still be able to understand a sexist joke about female programmers, or male nurses. We want an AI to be human, but we don't want it to associate CEOs with white males, dark hair, wearing suits. That will conflict.
- zozin 6y ago
  'authoritative data sources, such as Wikipedia" Lol
  - confeit 6y ago
    > Canberra is the capital city of Australia.

tda 6y ago

> It would be a bit like how carbon dating or production of low background steel changed after 1945 due to nuclear testing https://news.ycombinator.com/item?id=23896293 https://news.ycombinator.com/item?id=23896293 I guess books published will be more useful than reddit rants (depending on the application)

deleted 6y ago

[deleted]

lordnacho 6y ago

I've been wondering about something similar to you, but I read one of Pearl's causality books recently and thought that might be the missing piece. It's certainly impressive what GPT-3 can do, but it boggles the mind how much data went into it. By contrast a well-educated renaissance man might have read a book every month or so from age 15 to 30? That doesn't seem to be anywhere near what GPT could swallow in a few seconds. When you look at how GPT answers things, it kinda feels like someone who has heard the keywords and can spout some things that at least obscure whether it has ever studied a given subject, and this is impressive. What I wonder is whether it can do reasoning of the causality kind: what if X hadn't happened, what evidence do we need to collect to know if theory Z is falsified, which data W is confounding? To me it seems that sort of thing is what smart people are able to work out, with a lot of reading, but not quite the mountain that GPT reads.

skulk 6y ago
> By contrast a well-educated renaissance man might have read a book every month or so from age 15 to 30? That doesn't seem to be anywhere near what GPT could swallow in a few seconds. You're ignoring the insane amount of sensory information a human gets in 30 years. I think that absolutely dwarfs the amount of information that GPT-3 eats in a training run.
- lordnacho 6y ago
  That's an interesting point. I'm not sure how to measure that though. Also my guess is we have the sensors on for part of the day only, plus that's filtered heavily by your attention process, eg you can't read two books at once.
- PaulDavisThe1st 6y ago
  But that sensory information includes very few written words. GPT(n) isn't being trained on "worldly audio data", or "worldly tactile data", or in fact any sensory data at all. So the two training sets are completely orthogonal, and the well educated renaissance man is somehow able to take a very small exposure to written words and do at least as well as GPT(n) in processing them and responding.
  - klipt 6y ago
    Did this renaissance man teach themselves to read from scratch or are we assuming they were assisted in their schooling?
    - PaulDavisThe1st 6y ago
      Doesn't make too much difference to the overall point.
  - MauranKilom 6y ago
    And the renaissance man has tons of structure encoded in his brain on birth already. Just like GPT-3 does before you give it a prompt. I'm not saying this is fully equivalent (clearly a baby can't spout correct Latex just by seeing three samples), but you simply cannot just handwave away thousands of years of human evolution and millions of years of general evolution before that. The renaissance man is very obviously not working solely based on a few years of reading books (or learning to speak/write).

Veedrac 6y ago

> It cannot tell right from wrong It doesn't know which facts about the world are true and which are fabricated except through the text it's trained on, but to a large extent neither do you. It suffices to merely have the ability to reason about it. The primary difference is that whereas you reason fairly competently from one perspective with one pseudo-coherent set of goals, GPT-3 reasons to a weak degree from all perspectives, privileging no view point in particular. How important this ends up being depends on where the model plateaus. On one extreme, if it plateaus close to where GPT-3 already is, no harm done, it's a fun toy. On the other, if it scales until perplexity gets to far-superhuman levels, it doesn't matter at all, since you can just prompt it with Terence Tao talking about his latest discovery. Naturally, it will land somewhere in the middle of these points. The question is ultimately then whether the landing point captures enough general reasoning that you can use it to bootstrap some more advanced reasoning agent. A sufficiently powerful GPT-N should, for example, be able to deliberate over its own generated ideas and sort the coherent reasoning from the incoherent.

rytill 6y ago

I've also been wondering this. It is like Kessler Syndrome. We have to be careful not to pollute our ecosystem of data. https://en.wikipedia.org/wiki/Kessler_syndrome https://en.wikipedia.org/wiki/Kessler_syndrome Speaking with GPT-3 also makes one realize how influenced its predictions of AI scenarios are by dystopian memes. In any conversation in which you are "speaking to the AI", the AI can go rogue. There just aren't enough positive role models authors have written for AGI.

deleted 6y ago

[deleted]

simonh 6y ago

I don't think the problem is really with the data set, it's that GTP-3 doesn't actually have any understanding of the data. It's building a model of the text input to generate text output. It's not building a model of what the data means, or what it represents. When GTP-3 writes a scientific paper it's not trying to test a premise or critically evaluate some data, it's trying to generate text that looks like that sort of thing. It doesn't matter how many excellent quality papers you fed it, or how stringently you excluded low quality input data, it would still only be trying to produce facsimiles. It wouldn't be actually trying to do the things a real conscientious scientist is trying to do when they write a paper. Arguably at best it might be trying to do what a deceitful scientist trying to get credit for a paper with spurious results with no actual scientific merit might be doing when writing a paper that looks plausible, but thats actually a completely different activity. The code generation examples are really interesting. Here it's being used to generate real working code that has actual value, and it appears to work pretty well. It's code often has bugs, but who's doesn't? It only works for fairly short precisely definable coding tasks though. I don't think scaling it up to more complex coding problems is going to work. Again it doesn't understand the meaning of anything. It's trying to produce code that looks like working code, not actually solve the programming task you're giving it. It doesn't even know what a program is or what a programming task is. It doesn't know what input and output are. For example you can't ask it to modify existing code to change it's behaviour. It doesn't know code has behaviour. It has no idea what that even means and has no way to find out or any route to gaining that capability, because that's not a text transformation task and all it does is transform text.

SpicyLemonZest 6y ago
How can we tell whether or not GPT-3 has understanding of the data?
- deleted 6y ago
  [deleted]
- grugagag 6y ago
  I think the best way to tell whether GPT-3 has understanding of its data is by by asking questions related to the data but not explicit in the training dataset.

giancarlostoro 6y ago

> It seems to me that anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? Ah one only needs to think back to Tay to know how these sort of things will end. https://en.wikipedia.org/wiki/Tay_(bot) https://en.wikipedia.org/wiki/Tay_(bot) (Imagine 4chan got a wind of this bot and retrained it, which is probably what happened...)

Smaug123 6y ago

> it will just make up convincing rubbish It's not certain that this is always the case. In at least one case I've seen, if you give it question-and-answer prompts where you don't demonstrate that you will accept the answer "your question is nonsense", it will indeed make things up; but if you include "your question is nonsense" as an acceptable answer in the sample prompts, then it will use it correctly. See https://twitter.com/nicklovescode/status/1284050958977130497 https://twitter.com/nicklovescode/status/1284050958977130497 . It seems that we have a lot to learn about how to use GPT-3 effectively!

deleted 6y ago

[deleted]

resu_nimda 6y ago

It seems to me that anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? That same sentiment could be equally applied to humans, and not just in the internet era, but throughout all of history. There will always be misinformation and "wrong" opinions out there. "It cannot tell right from wrong" is an accusation leveled against human beings every day. We can't even all agree on what is right and wrong, truth or untruth. A true AI is going to have to wade through all that and make its own decisions to be viewed and judged from many different perspectives, just like the rest of us.

confeit 6y ago

> anything trained on internet data is kinda doomed to poison itself on the high ratio of garbage floating around here? Low-quality noise cancels out and leaves the high-quality signal. In the limit, the internet offers the true sequence probabilities for compression of natural text. You can also put more weight on authoritative data sources, such as Wikipedia and StackOverflow, but even uniformly weighted: It is possible to sequence-complete prime numbers, despite the many many pages online with random numbers. GPT-3 is trained on a filtered version of Common Crawl, enhanced with authoritative datasets, such as Books1, WebText, and Wikipedia-en. Moderation is done automatically, with a toxicity classifier/toggle. If GPT-n becomes good enough to be accepted in authoritative datasets, then it is perfectly fine training data, a form of semi-supervised learning. Bias is going to be a double-edged sword: I believe it will be impossible to prescribe common sense, nor to sanitize common sense to remove, say, gender bias, and still be able to understand a sexist joke about female programmers, or male nurses. We want an AI to be human, but we don't want it to associate CEOs with white males, dark hair, wearing suits. That will conflict.

zozin 6y ago
'authoritative data sources, such as Wikipedia" Lol
- confeit 6y ago
  > Canberra is the capital city of Australia.