8 ms·

Yeah, this seems like more of a problem than vague statements about AGI. We're still in the scope of ML - ChatGPT can't play chess, for example, and a self driv

Yeah, this seems like more of a problem than vague statements about AGI. We're still in the scope of ML - ChatGPT can't play chess, for example, and a self driving model can't write Haiku. An AGI would be able to do all of them. It seems much more likely that a fleet of autonomous (in the name of cutting costs) war machines will be created with relatively simple ML models that work in intended (or otherwise) ways to cause a lot of problems.

pmoriarty 3y ago
An AI doesn't have to be good in every domain to outcompete humans in many domains (which AI's already do). Besides, AI's can farm work out to other systems to do what they need. This has already been shown to work in practice with existing systems that do this.
- akira2501 3y ago
  Sure.. it also has to have sensors that outperform humans and require very little maintenance, or is maintenance that the platform can do itself without any renewable items, or with items that it can craft itself. It has to have a power density which outperforms humans with a reliable source of charging that it can defend, maintain and repair if necessary. Fatigue is an issue here too, if you can cause the machine to use more power in an interval than it could possibly regain by charging in that same interval, you've got them entirely on the back foot. Communications are an issue. Reliable distributed RF communications requires constant network coordination. These would be particularly easy to disrupt and for transmissions to be highly frustrated. Finally.. we have nuclear weapons. Detente is a real strategy.
lucumo 3y ago
> ChatGPT can't play chess, for example Out of pure curiosity I tried this a while back. At some point it tried to move a knight from d4 to f6, and it was the most stubborn and obstinate child when I tried to correct it. It was the best fun I had playing chess in a few decades. Excerpt: _Me:_ Can a knight move from f6 to d4? _ChatGPT:_ Yes, a knight can move from f6 to d4, as long as there are no pieces blocking its path and the square it is moving to is not occupied by one of its own pieces. In fact, the move Nxd4 that I suggested earlier involves moving a knight from its starting square to d4, capturing a pawn in the process. _Me:_ Which fields can a knight move to from f6? _ChatGPT:_ From f6, a knight can move to six different squares. These squares are: [... lists: d7, e8, g8, h7, d5, e4 ...] [...] _Me:_ You forgot g4 and h5, but no matter. Is d4 in that list? _ChatGPT:_ [...] To answer your other question, no, the square d4 is not one of the squares that a knight on f6 can move to directly. However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.
- kgwgk 3y ago
  > However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5. GPT stands for Goal Post Transportation.
- euroderf 3y ago
  So is this how hallucinations form ? It proposes something half-baked, and then later refers back to it as fact ?
  - abraxas 3y ago
    Sounds like we already have full blown cyber replicas of politicians then.
    - mythrwy 3y ago
      Or some bosses I've had. I actually can't wait to start reporting to GPT, how much crazier can it be?
  - zerocrates 3y ago
    Some of this is probably just an artifact of how ChatGPT specifically works: I believe I have it correct that it basically feeds the transcript of the conversation, to the extent possible, back to itself as part of the prompt going forward. So its prior responses in the session are part of the text it's generating from.
  - HarHarVeryFunny 3y ago
    No, it's generating output word by word, not planning very far ahead (it can't since it doesn't even know what words it's really going to generate, since they are randomly sampled), and essentially backs itself into a corner where completing the train of thought requires a fact that it doesn't actually have. Just as a made up example, say you asked "what is the capital city of england", and the model had seen similar questions in it's training data that were answered with "the capital city of X is Y", so it starts word-by-word generating this type of response "the capital city of england is", but it then turns out the model doesn't actually know the answer (i.e. this partial response context doesn't cause it to predict the correct answer), so it blithley predicts the next word as whatever it's inner machinations come up with, maybe "buckingham palace" or "flavor town". "Hallucination" seems a poor way to describe it, nor is it lying since there's no bad intent ... it's basically "starting to speak before brain engaged", a bit like a game show contestant being a bit too enthusiastic and hitting the "answer" button without actually having a fully thought out answer in mind.
jstanley 3y ago
I played chess against ChatGPT just yesterday, and it got into a winning position against me. After 24 moves it tried to play an illegal move and then when I told it that's illegal it played a bad move, and after that it didn't manage to find any more legal moves (I gave up after asking it to try again about 10 times). But it's very close to being able to play chess. My prompt was: > we're playing chess. only send chess moves. > > 1. e4
- zimpenfish 3y ago
  What does it do if you play illegal moves?
  - jstanley 3y ago
    Good idea, I tried that just now. It just accepts it and carries on.
    - TedDoesntTalk 3y ago
      It does the same when you ask it to be DM in a D&D game. It allows the players to do many, many things outside the rules. I don't remember any examples but a general idea was, "The character Frodo now has a the ability to breath fire. He breathes fire on the orcs." Although IIRC that was ChatGPT 3.5.
      - sebzim4500 3y ago
        I wonder if this would happen if you used the GPT-4 api with a system message. In my experience that can help but does necessarily solve the issue.
      - Closi 3y ago
        You can overcome this with prompt engineering. “If a player tries to do something not strictly within the rules of <insert game>, then you must inform me that it is an invalid move and not accept it” GPT appears to be slightly tuned to default to “yes and” in these ‘creative’ situations by default rather than “block/deny”. IMO lots of things where people don’t think GPT can do something end up being possible with basic prompt engineering. Usually people go too-short and too-non-specific on the prompt.
sangnoir 3y ago
I'm more worried about unethical applications of current ML SOTA than AGI. A trivial example would be removing humans out of the loop in antipersonnel armed drones so they autonomously classify targets and if/when to pull the trigger.
endtime 3y ago
> ChatGPT can't play chess, for example Do you really expect this claim to age well? I wouldn't be at all surprised if GPT5 could play chess quite well. The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it, but the worst possible time to start worrying about it is when we unquestionably have it. Because the whole point is that that is too late.
- pmoriarty 3y ago
  "The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it" He was far, far from the first. Warnings about the threat of intelligent machines have been around for over 100 years. The first instance I know of is E. M. Foster's "The Machine Stops"[1]. which was published in 1909, though I wouldn't be surprised if there were even earlier warnings. They've been well worn scifi tropes ever since. [1] - https://en.m.wikipedia.org/wiki/The_Machine_Stops https://en.m.wikipedia.org/wiki/The_Machine_Stops
  - ben_w 3y ago
    I'd say Frankenstein, and before that magical "didn't think it though" errors like The Sorcerer's Apprentice or Midas, or "didn't read the instructions" errors such as Grottasöngr[0] or 水母娘娘 and the magic whip[1]. That said, Yudkowsky seems (rightly or wrongly) to get the credit for turning this from merely fun stories into a serious (and not insane) field of study. [0] https://en.wikipedia.org/wiki/Grottasöngr https://en.wikipedia.org/wiki/Grottasöngr [1] https://en.wikipedia.org/wiki/Shuimu https://en.wikipedia.org/wiki/Shuimu
  - endtime 3y ago
    There's been a lot of sci-fi, I agree. I see that as quite distinct from Yudkowsky's efforts.
rst 3y ago
People have gotten it to play chess. The quality of play you get is very sensitive to prompt details, and may also be affected by OpenAI's ongoing model tuning, but here's one experiment in which it won 11 games of 19 against amateurs (treating any illegal move it tried to make as an immediate forfeit): https://dkb.blog/p/chatgpts-chess-elo-is-1400 https://dkb.blog/p/chatgpts-chess-elo-is-1400
HarHarVeryFunny 3y ago
It's not surprising that ChatGPT can't play chess well, since: 1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion. It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good. 2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much. 3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.

pmoriarty 3y ago

An AI doesn't have to be good in every domain to outcompete humans in many domains (which AI's already do). Besides, AI's can farm work out to other systems to do what they need. This has already been shown to work in practice with existing systems that do this.

akira2501 3y ago
Sure.. it also has to have sensors that outperform humans and require very little maintenance, or is maintenance that the platform can do itself without any renewable items, or with items that it can craft itself. It has to have a power density which outperforms humans with a reliable source of charging that it can defend, maintain and repair if necessary. Fatigue is an issue here too, if you can cause the machine to use more power in an interval than it could possibly regain by charging in that same interval, you've got them entirely on the back foot. Communications are an issue. Reliable distributed RF communications requires constant network coordination. These would be particularly easy to disrupt and for transmissions to be highly frustrated. Finally.. we have nuclear weapons. Detente is a real strategy.

lucumo 3y ago

> ChatGPT can't play chess, for example Out of pure curiosity I tried this a while back. At some point it tried to move a knight from d4 to f6, and it was the most stubborn and obstinate child when I tried to correct it. It was the best fun I had playing chess in a few decades. Excerpt: _Me:_ Can a knight move from f6 to d4? _ChatGPT:_ Yes, a knight can move from f6 to d4, as long as there are no pieces blocking its path and the square it is moving to is not occupied by one of its own pieces. In fact, the move Nxd4 that I suggested earlier involves moving a knight from its starting square to d4, capturing a pawn in the process. _Me:_ Which fields can a knight move to from f6? _ChatGPT:_ From f6, a knight can move to six different squares. These squares are: [... lists: d7, e8, g8, h7, d5, e4 ...] [...] _Me:_ You forgot g4 and h5, but no matter. Is d4 in that list? _ChatGPT:_ [...] To answer your other question, no, the square d4 is not one of the squares that a knight on f6 can move to directly. However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.

kgwgk 3y ago
> However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5. GPT stands for Goal Post Transportation.
euroderf 3y ago
So is this how hallucinations form ? It proposes something half-baked, and then later refers back to it as fact ?
- abraxas 3y ago
  Sounds like we already have full blown cyber replicas of politicians then.
  - mythrwy 3y ago
    Or some bosses I've had. I actually can't wait to start reporting to GPT, how much crazier can it be?
- zerocrates 3y ago
  Some of this is probably just an artifact of how ChatGPT specifically works: I believe I have it correct that it basically feeds the transcript of the conversation, to the extent possible, back to itself as part of the prompt going forward. So its prior responses in the session are part of the text it's generating from.
- HarHarVeryFunny 3y ago
  No, it's generating output word by word, not planning very far ahead (it can't since it doesn't even know what words it's really going to generate, since they are randomly sampled), and essentially backs itself into a corner where completing the train of thought requires a fact that it doesn't actually have. Just as a made up example, say you asked "what is the capital city of england", and the model had seen similar questions in it's training data that were answered with "the capital city of X is Y", so it starts word-by-word generating this type of response "the capital city of england is", but it then turns out the model doesn't actually know the answer (i.e. this partial response context doesn't cause it to predict the correct answer), so it blithley predicts the next word as whatever it's inner machinations come up with, maybe "buckingham palace" or "flavor town". "Hallucination" seems a poor way to describe it, nor is it lying since there's no bad intent ... it's basically "starting to speak before brain engaged", a bit like a game show contestant being a bit too enthusiastic and hitting the "answer" button without actually having a fully thought out answer in mind.

jstanley 3y ago

I played chess against ChatGPT just yesterday, and it got into a winning position against me. After 24 moves it tried to play an illegal move and then when I told it that's illegal it played a bad move, and after that it didn't manage to find any more legal moves (I gave up after asking it to try again about 10 times). But it's very close to being able to play chess. My prompt was: > we're playing chess. only send chess moves. > > 1. e4

zimpenfish 3y ago
What does it do if you play illegal moves?
- jstanley 3y ago
  Good idea, I tried that just now. It just accepts it and carries on.
  - TedDoesntTalk 3y ago
    It does the same when you ask it to be DM in a D&D game. It allows the players to do many, many things outside the rules. I don't remember any examples but a general idea was, "The character Frodo now has a the ability to breath fire. He breathes fire on the orcs." Although IIRC that was ChatGPT 3.5.
    - sebzim4500 3y ago
      I wonder if this would happen if you used the GPT-4 api with a system message. In my experience that can help but does necessarily solve the issue.
    - Closi 3y ago
      You can overcome this with prompt engineering. “If a player tries to do something not strictly within the rules of <insert game>, then you must inform me that it is an invalid move and not accept it” GPT appears to be slightly tuned to default to “yes and” in these ‘creative’ situations by default rather than “block/deny”. IMO lots of things where people don’t think GPT can do something end up being possible with basic prompt engineering. Usually people go too-short and too-non-specific on the prompt.

sangnoir 3y ago

I'm more worried about unethical applications of current ML SOTA than AGI. A trivial example would be removing humans out of the loop in antipersonnel armed drones so they autonomously classify targets and if/when to pull the trigger.

endtime 3y ago

> ChatGPT can't play chess, for example Do you really expect this claim to age well? I wouldn't be at all surprised if GPT5 could play chess quite well. The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it, but the worst possible time to start worrying about it is when we unquestionably have it. Because the whole point is that that is too late.

pmoriarty 3y ago
"The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it" He was far, far from the first. Warnings about the threat of intelligent machines have been around for over 100 years. The first instance I know of is E. M. Foster's "The Machine Stops"[1]. which was published in 1909, though I wouldn't be surprised if there were even earlier warnings. They've been well worn scifi tropes ever since. [1] - https://en.m.wikipedia.org/wiki/The_Machine_Stops https://en.m.wikipedia.org/wiki/The_Machine_Stops
- ben_w 3y ago
  I'd say Frankenstein, and before that magical "didn't think it though" errors like The Sorcerer's Apprentice or Midas, or "didn't read the instructions" errors such as Grottasöngr[0] or 水母娘娘 and the magic whip[1]. That said, Yudkowsky seems (rightly or wrongly) to get the credit for turning this from merely fun stories into a serious (and not insane) field of study. [0] https://en.wikipedia.org/wiki/Grottasöngr https://en.wikipedia.org/wiki/Grottasöngr [1] https://en.wikipedia.org/wiki/Shuimu https://en.wikipedia.org/wiki/Shuimu
- endtime 3y ago
  There's been a lot of sci-fi, I agree. I see that as quite distinct from Yudkowsky's efforts.

rst 3y ago

People have gotten it to play chess. The quality of play you get is very sensitive to prompt details, and may also be affected by OpenAI's ongoing model tuning, but here's one experiment in which it won 11 games of 19 against amateurs (treating any illegal move it tried to make as an immediate forfeit): https://dkb.blog/p/chatgpts-chess-elo-is-1400 https://dkb.blog/p/chatgpts-chess-elo-is-1400

HarHarVeryFunny 3y ago

It's not surprising that ChatGPT can't play chess well, since: 1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion. It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good. 2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much. 3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.