3 ms·

I don't think AI has to be low-quality for GP's concern to be valid. Humans get inputs from a large variety of sources, but if an AI's input is just text, then

I don't think AI has to be low-quality for GP's concern to be valid. Humans get inputs from a large variety of sources, but if an AI's input is just text, then there's the potential for AI's input to mostly consist of its prior output. Iterate this, and its model could gradually diverge from the real world. The equivalent in human society is groupthink, where members of a subgroup get most of their information from the same subgroup and end up believing weird things. We can counter that by purposely getting inputs from outside of our group. For a text-model AI, this means identifying text that wasn't produced by AI, as the article suggests.

ben_w 4y ago
Kinda? AI is often limited in ways we aren't, but it also trivially consumes more than we can in a lifetime. "Just text" in the case of GPT-3, but also it is trained on a token count exceeding the number of times an average synapse in a human brain will fire in a lifetime. It can still get biases from the training set; while I'm not sure if "group think" is quite the right phrase, it does seem to "want" everyone to get along even when asking it to create multiple characters engaged in a conflict. (Or perhaps that's just an artefact of it estimating that I want that). Reminds me — in a bad way — of Jules Verne's From The Earth to the Moon.
- DennisP 4y ago
  Consuming and producing vast amounts of information is what makes the problem potentially worse than human groupthink. It enables the situation where AI is mostly consuming information produced by AI. That's the feedback loop I'm calling "groupthink." It could end up diverging from reality in the same way that chaotic functions diverge widely due to tiny differences in the initial conditions. The same problem exists if the AI consumes other types of information that it also produces. Humans are more grounded by having a presence in the physical world. Plus they draw on various sources considered more reliable, like formal training, scientific papers, textbooks, quality journalism, etc. If we want AI to be reliable, we'll need it to put the most weight on similar sources, and maybe even have some real-world presence with sensors and robots. Eventually AI will be able to produce new reliable information itself. But for that, it would have to recognize factual inconsistencies between sources and logical inconsistencies in arguments, and figure out how to resolve those, and do math correctly. I don't know what the state of the art is here, though ChatGPT tends to fail at basic arithmetic.
  - ben_w 4y ago
    > That's the feedback loop I'm calling "groupthink." While I think I get your point, I'd call that failure mode "believing it's own BS", and (perhaps I'm just being cynical here) I think humans collectively also have this failure mode. That said, there's an old saying: "To err is human, to really foul up requires a computer" — it is quite possible for a machine, with merely the same category of flaws we have and no others, to be really bad for the world just because it's really fast and doesn't sleep.
joshspankit 4y ago
There’s nothing saying AIs have to have text input, it’s just the method with the lowest friction of imagination. That’s why books of text have been around for so long. There are already AIs that take input via image, video, and audio. The AI tech is input agnostic and only requires that someone figures out a way to get the input in.
- DennisP 4y ago
  Yes but the issue isn't specific to text. It works for anything that AI also produces.

ben_w 4y ago

Kinda? AI is often limited in ways we aren't, but it also trivially consumes more than we can in a lifetime. "Just text" in the case of GPT-3, but also it is trained on a token count exceeding the number of times an average synapse in a human brain will fire in a lifetime. It can still get biases from the training set; while I'm not sure if "group think" is quite the right phrase, it does seem to "want" everyone to get along even when asking it to create multiple characters engaged in a conflict. (Or perhaps that's just an artefact of it estimating that I want that). Reminds me — in a bad way — of Jules Verne's From The Earth to the Moon.

DennisP 4y ago
Consuming and producing vast amounts of information is what makes the problem potentially worse than human groupthink. It enables the situation where AI is mostly consuming information produced by AI. That's the feedback loop I'm calling "groupthink." It could end up diverging from reality in the same way that chaotic functions diverge widely due to tiny differences in the initial conditions. The same problem exists if the AI consumes other types of information that it also produces. Humans are more grounded by having a presence in the physical world. Plus they draw on various sources considered more reliable, like formal training, scientific papers, textbooks, quality journalism, etc. If we want AI to be reliable, we'll need it to put the most weight on similar sources, and maybe even have some real-world presence with sensors and robots. Eventually AI will be able to produce new reliable information itself. But for that, it would have to recognize factual inconsistencies between sources and logical inconsistencies in arguments, and figure out how to resolve those, and do math correctly. I don't know what the state of the art is here, though ChatGPT tends to fail at basic arithmetic.
- ben_w 4y ago
  > That's the feedback loop I'm calling "groupthink." While I think I get your point, I'd call that failure mode "believing it's own BS", and (perhaps I'm just being cynical here) I think humans collectively also have this failure mode. That said, there's an old saying: "To err is human, to really foul up requires a computer" — it is quite possible for a machine, with merely the same category of flaws we have and no others, to be really bad for the world just because it's really fast and doesn't sleep.

joshspankit 4y ago

There’s nothing saying AIs have to have text input, it’s just the method with the lowest friction of imagination. That’s why books of text have been around for so long. There are already AIs that take input via image, video, and audio. The AI tech is input agnostic and only requires that someone figures out a way to get the input in.

DennisP 4y ago
Yes but the issue isn't specific to text. It works for anything that AI also produces.