20 ms·

We are barely scraping the surface when it comes to understanding the future dangers of AI. Geoffrey Hinton is uniquely positioned to point out where the danger

We are barely scraping the surface when it comes to understanding the future dangers of AI. Geoffrey Hinton is uniquely positioned to point out where the dangers are, and from what I've gleaned from interviews one of his main concerns atm is the use of AI in the military: fully autonomous military robots might not be possible to curtail. The tried and tested method is international agreements. The current focus on arms race and militarily subduing enemies does not give much hope however. Still, global binding agreements are likely where the solution lies IMO, both in this case and others where some types of weapons are too dangerous to use, so let's not give up on that so easily.

sudhirj 3y ago
Yeah, this seems like more of a problem than vague statements about AGI. We're still in the scope of ML - ChatGPT can't play chess, for example, and a self driving model can't write Haiku. An AGI would be able to do all of them. It seems much more likely that a fleet of autonomous (in the name of cutting costs) war machines will be created with relatively simple ML models that work in intended (or otherwise) ways to cause a lot of problems.
- pmoriarty 3y ago
  An AI doesn't have to be good in every domain to outcompete humans in many domains (which AI's already do). Besides, AI's can farm work out to other systems to do what they need. This has already been shown to work in practice with existing systems that do this.
  - akira2501 3y ago
    Sure.. it also has to have sensors that outperform humans and require very little maintenance, or is maintenance that the platform can do itself without any renewable items, or with items that it can craft itself. It has to have a power density which outperforms humans with a reliable source of charging that it can defend, maintain and repair if necessary. Fatigue is an issue here too, if you can cause the machine to use more power in an interval than it could possibly regain by charging in that same interval, you've got them entirely on the back foot. Communications are an issue. Reliable distributed RF communications requires constant network coordination. These would be particularly easy to disrupt and for transmissions to be highly frustrated. Finally.. we have nuclear weapons. Detente is a real strategy.
- lucumo 3y ago
  > ChatGPT can't play chess, for example Out of pure curiosity I tried this a while back. At some point it tried to move a knight from d4 to f6, and it was the most stubborn and obstinate child when I tried to correct it. It was the best fun I had playing chess in a few decades. Excerpt: _Me:_ Can a knight move from f6 to d4? _ChatGPT:_ Yes, a knight can move from f6 to d4, as long as there are no pieces blocking its path and the square it is moving to is not occupied by one of its own pieces. In fact, the move Nxd4 that I suggested earlier involves moving a knight from its starting square to d4, capturing a pawn in the process. _Me:_ Which fields can a knight move to from f6? _ChatGPT:_ From f6, a knight can move to six different squares. These squares are: [... lists: d7, e8, g8, h7, d5, e4 ...] [...] _Me:_ You forgot g4 and h5, but no matter. Is d4 in that list? _ChatGPT:_ [...] To answer your other question, no, the square d4 is not one of the squares that a knight on f6 can move to directly. However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.
  - kgwgk 3y ago
    > However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5. GPT stands for Goal Post Transportation.
  - euroderf 3y ago
    So is this how hallucinations form ? It proposes something half-baked, and then later refers back to it as fact ?
    - abraxas 3y ago
      Sounds like we already have full blown cyber replicas of politicians then.
      - mythrwy 3y ago
        Or some bosses I've had. I actually can't wait to start reporting to GPT, how much crazier can it be?
    - 3y ago
- jstanley 3y ago
  I played chess against ChatGPT just yesterday, and it got into a winning position against me. After 24 moves it tried to play an illegal move and then when I told it that's illegal it played a bad move, and after that it didn't manage to find any more legal moves (I gave up after asking it to try again about 10 times). But it's very close to being able to play chess. My prompt was: > we're playing chess. only send chess moves. > > 1. e4
  - zimpenfish 3y ago
    What does it do if you play illegal moves?
    - jstanley 3y ago
      Good idea, I tried that just now. It just accepts it and carries on.
      - TedDoesntTalk 3y ago
        It does the same when you ask it to be DM in a D&D game. It allows the players to do many, many things outside the rules. I don't remember any examples but a general idea was, "The character Frodo now has a the ability to breath fire. He breathes fire on the orcs." Although IIRC that was ChatGPT 3.5.
        sebzim4500 3y ago
        I wonder if this would happen if you used the GPT-4 api with a system message. In my experience that can help but does necessarily solve the issue.
        Closi 3y ago
        You can overcome this with prompt engineering. “If a player tries to do something not strictly within the rules of <insert game>, then you must inform me that it is an invalid move and not accept it” GPT appears to be slightly tuned to default to “yes and” in these ‘creative’ situations by default rather than “block/deny”. IMO lots of things where people don’t think GPT can do something end up being possible with basic prompt engineering. Usually people go too-short and too-non-specific on the prompt.
- sangnoir 3y ago
  I'm more worried about unethical applications of current ML SOTA than AGI. A trivial example would be removing humans out of the loop in antipersonnel armed drones so they autonomously classify targets and if/when to pull the trigger.
- endtime 3y ago
  > ChatGPT can't play chess, for example Do you really expect this claim to age well? I wouldn't be at all surprised if GPT5 could play chess quite well. The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it, but the worst possible time to start worrying about it is when we unquestionably have it. Because the whole point is that that is too late.
  - pmoriarty 3y ago
    "The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it" He was far, far from the first. Warnings about the threat of intelligent machines have been around for over 100 years. The first instance I know of is E. M. Foster's "The Machine Stops"[1]. which was published in 1909, though I wouldn't be surprised if there were even earlier warnings. They've been well worn scifi tropes ever since. [1] - https://en.m.wikipedia.org/wiki/The_Machine_Stops https://en.m.wikipedia.org/wiki/The_Machine_Stops
    - ben_w 3y ago
      I'd say Frankenstein, and before that magical "didn't think it though" errors like The Sorcerer's Apprentice or Midas, or "didn't read the instructions" errors such as Grottasöngr[0] or 水母娘娘 and the magic whip[1]. That said, Yudkowsky seems (rightly or wrongly) to get the credit for turning this from merely fun stories into a serious (and not insane) field of study. [0] https://en.wikipedia.org/wiki/Grottasöngr https://en.wikipedia.org/wiki/Grottasöngr [1] https://en.wikipedia.org/wiki/Shuimu https://en.wikipedia.org/wiki/Shuimu
    - endtime 3y ago
      There's been a lot of sci-fi, I agree. I see that as quite distinct from Yudkowsky's efforts.
- rst 3y ago
  People have gotten it to play chess. The quality of play you get is very sensitive to prompt details, and may also be affected by OpenAI's ongoing model tuning, but here's one experiment in which it won 11 games of 19 against amateurs (treating any illegal move it tried to make as an immediate forfeit): https://dkb.blog/p/chatgpts-chess-elo-is-1400 https://dkb.blog/p/chatgpts-chess-elo-is-1400
- HarHarVeryFunny 3y ago
  It's not surprising that ChatGPT can't play chess well, since: 1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion. It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good. 2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much. 3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.
kranke155 3y ago
How would you curtail their use when any military that commits to using them will have a huge advantage ? This isn’t like nuclear weapons where any use is curtailed by the apocalyptic outcomes. Killer robots are the way we will fight in the future and any military which refuses to deploy them will find themselves facing defeat.
- throw_nbvc1234 3y ago
  AI is not allowed to make the decision to kill, that must be done by a human. Having every other step done by an AI is still going to be a huge advantage over non-AI military forces.
  - michaelmior 3y ago
    Even if the decision is made by a human, it really matters how that decision is presented. My understanding of the current state of warfare (which could be way off) is that were possible, targets are still largely selected manually based on verified intelligence. What if we allow AI to propose targets? Even if testing shows that the AI is highly effective at picking "good" targets, that changes the situation entirely. Suppose humans make the final decision of which target to strike. How much time is given to make that decision? What information is presented to them? If the answer is that humans make a decision in a short amount of time largely relying on an AI-generated confidence score, are humans really making the decision?
    - Keegs 3y ago
      Exactly. It’s too easy for these decisions to become a formality. ProPublica’s reporting on Cigna[1] is a great example: > Before health insurers reject claims for medical reasons, company doctors must review them, according to insurance laws > A Cigna algorithm flags mismatches between diagnoses and what the company considers acceptable tests and procedures for those ailments Sounds fine, except > “We literally click and submit,” one former Cigna doctor said. “It takes all of 10 seconds to do 50 at a time.” [1]: https://www.propublica.org/article/cigna-pxdx-medical-health-insurance-rejection-claims https://www.propublica.org/article/cigna-pxdx-medical-health...
lumost 3y ago
There is such a blurry line for autonomous munitions. militaries used dumb imprecise munitions for decades - then precision weapons. A2A missiles used to lock on radar signature leading to huge risks related to accidentally shooting airliners/friendly craft. Now antiship missiles dynamically select their target over 300km away to maximize the chance of hitting a big ship. During the war on terror, ML models would decide which phone to blow up. We’re probably going to see ai driven target selection and prioritization for fire control within the next few months of the Ukraine war. The US’s new Rapid dragon program almost demands ai control of target selection and flight trajectories. Where do you draw the line? What would an appropriate agreement look like?
- stareatgoats 3y ago
  > Where do you draw the line? What would an appropriate agreement look like? Good questions. I'm not sure, but that line needs to be drawn somewhere. Thousands of experts should be pouring over questions like these, and likely will.
  - JieJie 3y ago
    Here's an interview[0] with Lauren Khan from the Council on Foreign Relations[1] who is indeed pouring over those questions. [0]https://youtu.be/CDoIPpcwnCE https://youtu.be/CDoIPpcwnCE [1]https://www.cfr.org/expert/lauren-kahn https://www.cfr.org/expert/lauren-kahn
  - milesward 3y ago
    Who pays for those experts, and what are their incentives?
- neatze 3y ago
  The line already drawn, no indiscriminate killings of civilians, all this technology only minimizes civilian suffering, furthermore it is more humane to use terminators instead of 20 years olds clearing buildings and autonomous tanks/tracks driving though IEDs filled streets. War is deeply in human nature, it is not going away, question is; how to make it more humane and minimize indiscriminate killings and sufferings in most cases poorest populations on earth.
1024core 3y ago
> The tried and tested method is international agreements. You really think actors like North Korea, Al Qaeda, etc. will adhere to International agreements?!?
- flangola7 3y ago
  Against the combined militaries of the rest of the planet? Yes.
- waboremo 3y ago
  Yes, they adhere to many international agreements already. You can look up North Korea's signed treaties if you're unsure. Ignoring the Al Qaeda part (or similars) because a fragile extremist group barely held together is unlikely to have the resources to assemble anything devastating enough to warrant infringing on an international agreement regarding militarized AI.
- mtlmtlmtlmtl 3y ago
  I'm not terribly worried about Al Qaeda or DPRK having any sort of capability to develop planetkilling AI, now or in the future. Nukes and Rockets? Sure. Anyone can build a centrifuge, and anyone can build a cylinder with explosives on it. But intelligent killer robots? Please.
slashdev 3y ago
It’s not the war robots that worry me as much as centralized intelligence with internet connectivity. War robots don’t reproduce, require energy infrastructure, and can be destroyed. While they could run amok, by targeting things they’re not supposed to, they won’t really be intelligent because the problem doesn’t require much intelligence. Now if they’re controlled by a central intelligence that’s a bit scarier.
- Eisenstein 3y ago
  What if the only way to stop war from occurring is to entirely remove humans from the decision making that would cause it?
  - slashdev 3y ago
    You don’t get that with war robots, the decision to go to war or not is still in the hands of the politicians. But I wonder sometimes what it would take to end war.
uoaei 3y ago
I can't really tell if he's had a sincere change of heart about it. Certainly his screeds about how DL is the only path forward for AGI rang extremely hollow even 2 or 3 years ago. Those comments were clearly motivated by profit, considering his position in the field and all the companies vying for him at the time.
ecnahc515 3y ago
Let's hope we don't get to Horizon Zero Dawn too soon.
gumballindie 3y ago
An EMP bomb can easily sort out robots but nothing can protect us from data and ip theft. That’s the real danger here unless regulated quickly.
ryan93 3y ago
No one is uniquely positioned. Literally no one knows how powerful it will get.
O5vYtytb 3y ago
My biggest concern for military use of AI is how incompetent most military contractors are. These huge companies employ an army of not-very-good engineers whose primary purpose seems to be to over-complicate projects. Imagine the same teams that make planes which need to be hard rebooted every few days, now they're making advanced AI to dynamically target and kill people.
- dmix 3y ago
  > now they're making advanced AI to dynamically target and kill people. I'm curious, which projects are working on this? Any ones that we know about? In terms of production weapons I know the newer Switchblades, Excalibur, and even Javelin do some fancy automated targeting before the terminal phase but that's not really AI beyond image recognition and the targets/specific destination are pre-selected. I'm curious what sort of applications would use autonomous targeting without overt human involvement.
  - ethbr0 3y ago
    Air defense has had automatic targeting since the 1980s(?) due to the required reaction times. As for autonomous target/kill programs, that'd be the USAF's Collaborative Combat Aircraft (CCA) program (under NGAD). It sounds like they're keeping human-in-the-loop for now by targeting a 1:many manned_stealth:drone ratio, but that will be subject to erosion as autonomous capabilities improve. And the very nature of the mission requires autonomy -- drones penetrating deep into sophisticated air defense systems using stealth and emissions control (so only line-of-sight communication links). Nobody is going to be remotely piloting these things.
  - ikekkdcjkfke 3y ago
    You can bet they have automatic target tracking and ballistic aiming onto those in the new abrams tank, but sending the kill command will most likely be done by operators. But it's just a small step from that to putting it on autopilot
    - dmix 3y ago
nobodyandproud 3y ago
Outcome: Automate the economy, and employ the dispossessed to kill one another in the name of ethics (because AI military is unethical). This seems weird and arbitrary.
ren_engineer 3y ago
Military application of AI drones isn't even the worst possible use, it's nations using them to completely subjugate their own population(although the same tech could be used against non-peer nations). Combination of things like Gorgon Stare to direct smaller AI controlled drones like what they are using in Ukraine would be a police state nightmare. https://en.wikipedia.org/wiki/Gorgon_Stare https://en.wikipedia.org/wiki/Gorgon_Stare https://longreads.com/2019/06/21/nothing-kept-me-up-at-night-the-way-the-gorgon-stare-did/ https://longreads.com/2019/06/21/nothing-kept-me-up-at-night... they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with. No form of protest or movement against the government will be possible if it's scaled up
- bluetwo 3y ago
  At a music festival last summer I counted 4 drones in the sky monitoring 24/7 over 5 days. Never saw them come down. They were each tethered to a base station which seemed to run electric and probably returned a video feed. I expect to see this everywhere.
  - senbrow 3y ago
    Which festival? And where?
  - heavyset_go 3y ago
    The last few years has had the police flying drones over events where I am. Think like fairs, parades, etc.
- l3mure 3y ago
  Yup. [1] > Pentagon testing mass surveillance balloons across the US [2] > For instance, using Gorgon Stare, a 24-hour aerial surveillance system, the U.S. Air Force had been able to plot back from a car bomb explosion in Kabul in 2019, which killed 126 civilians, to find the location of safe houses used to execute the attack. [1] - https://www.theguardian.com/us-news/2019/aug/02/pentagon-balloons-surveillance-midwest https://www.theguardian.com/us-news/2019/aug/02/pentagon-bal... [2] - https://warontherocks.com/2023/04/ai-at-war/ https://warontherocks.com/2023/04/ai-at-war/
deskamess 3y ago
International treaties can hold to an extent. The greatest damage will be its internal use. Where countries can tell others to 'not interfere' in local business. Each country will run its own nefarious program and it will take a violent revolution to overthrow governments - and the next one will pick up the AI baton where the previous one left with a slogan of 'making sure no one does what the previous govt did'. So instead of an international global AI issue we will have strong national AI abuse. In either case, democracy will be put under strain.
dukeofdoom 3y ago
Leading theory is that COVID was made in a lab. Not sure what to fear more AI robots, or AI engineered viruses.
deleted 3y ago
[deleted]
heavyset_go 3y ago
Black Mirror got it right in their "Metalhead" episode, which is probably my favorite.
nradov 3y ago
International agreements are hardly tried and tested. The Nonproliferation Treaty has been somewhat effective with nuclear weapons largely because refining operations are hard to hide, and even with that several additional countries have acquired such weapons. Agreements on chemical and biological weapons are largely moot because it turns out that such weapons aren't even very effective compared to kinetic alternatives. The ban on land mines was never ratified by the countries that do most fighting, and such mines are being heavily used by both sides in Ukraine. The Washington Naval Treaty was a total failure. The ban on space weapons is breaking down right now. It is impossible to have an effective international agreement on autonomous weapons. No military power would ever agree to let a third party inspect their weapon source code in a verifiable way. It's too easy to hide the real code, and we would never trust potential adversaries not to cheat. Fully autonomous weapons have already been deployed for decades. The Mark 60 CAPTOR mine could sit and wait for weeks until it detected a probable target matching a programmed signature, then launch a homing torpedo at it. After the initial deployment there is no human in the loop.
- stareatgoats 3y ago
  FWIW, you make a good point, albeit a pessimistic one - at least if you like me dread the chicken race towards doom that is the obvious alternative. Agree that international agreements may not always have been the success stories I implied. But even the fact that some states have not signed certain treaties prove that they are effective (otherwise they could have signed and then just ignored them). Once a state has signed a treaty you won't need to rely on outside inspectors exclusively - inside whistleblowers will do just fine.
  - nradov 3y ago
    How many inside whistleblowers do you think we will get from China, North Korea, and Iran?
    - stareatgoats 3y ago
      Whistleblowers, spies, side-channel intelligence. Not that many tools but enough to make most states avoid entering agreements they won't honor. No state wants to be exposed as a deceptive liar, it goes to their legitimacy. Impossible? well believe what you like. I'd like to believe that it is possible.
dan-robertson 3y ago
Sure, his position is reasonably unique, and he’s potentially had a broad overview of lots of things going on at Google and the industry in general, but is your claim that he is good at pointing out dangers because he hears lots of gossip, or is it that being involved in deep learning for a long time makes him good at figuring out those things. I definitely don’t buy the latter. What, precisely, is the reason you think Hinton would be good at pointing out dangers? Maybe you just mean that journalists will be happy to interview him rather than that he is likely to be right? Certainly that does give one an advantage in pointing things out.
api 3y ago
The scenario that I find both most scary and most likely is the use of AI to propagandize, brainwash, and con human beings at scale. Basically you can now assign every single living human being their own 24/7 con artist and power that con artist with reams of personalized surveillance information about each target purchased from data brokers. Everyone will have a highly informed personalized con artist following them around 24/7 trying to convince them of whatever the controller of that bot has programmed it to sell. We're creating the propaganda equivalent of the hydrogen bomb.
ericmcer 3y ago
Threats like this seem less real to me because the government has been so technologically inept lately. Garbage government websites, failed rollouts of huge programs (like healthcare, the CA highspeed rail), SpaceX taking the reigns away from NASA and the military awarding giant contracts to Amazon and Microsoft to keep their ancient tech infra running. It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them.
- DrBenCarson 3y ago
  Um yeah...someone sells the government all of its weapons. Literally all of them. It would be a dramatic change for the government to in-source weapons development.
- alecbz 3y ago
  I don't know too much about the internals of the military, but I think they already have a lot of very impressive technology that would seem at-odds with other signs of governmental ineptitude with technology. A government website being shit might not be viewed with as much urgency and importance as combat technology. > It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them. I mean, that might be how it happens? Military contractors might as well be seen as a part of the military.

sudhirj 3y ago

Yeah, this seems like more of a problem than vague statements about AGI. We're still in the scope of ML - ChatGPT can't play chess, for example, and a self driving model can't write Haiku. An AGI would be able to do all of them. It seems much more likely that a fleet of autonomous (in the name of cutting costs) war machines will be created with relatively simple ML models that work in intended (or otherwise) ways to cause a lot of problems.

pmoriarty 3y ago
An AI doesn't have to be good in every domain to outcompete humans in many domains (which AI's already do). Besides, AI's can farm work out to other systems to do what they need. This has already been shown to work in practice with existing systems that do this.
- akira2501 3y ago
  Sure.. it also has to have sensors that outperform humans and require very little maintenance, or is maintenance that the platform can do itself without any renewable items, or with items that it can craft itself. It has to have a power density which outperforms humans with a reliable source of charging that it can defend, maintain and repair if necessary. Fatigue is an issue here too, if you can cause the machine to use more power in an interval than it could possibly regain by charging in that same interval, you've got them entirely on the back foot. Communications are an issue. Reliable distributed RF communications requires constant network coordination. These would be particularly easy to disrupt and for transmissions to be highly frustrated. Finally.. we have nuclear weapons. Detente is a real strategy.
lucumo 3y ago
> ChatGPT can't play chess, for example Out of pure curiosity I tried this a while back. At some point it tried to move a knight from d4 to f6, and it was the most stubborn and obstinate child when I tried to correct it. It was the best fun I had playing chess in a few decades. Excerpt: _Me:_ Can a knight move from f6 to d4? _ChatGPT:_ Yes, a knight can move from f6 to d4, as long as there are no pieces blocking its path and the square it is moving to is not occupied by one of its own pieces. In fact, the move Nxd4 that I suggested earlier involves moving a knight from its starting square to d4, capturing a pawn in the process. _Me:_ Which fields can a knight move to from f6? _ChatGPT:_ From f6, a knight can move to six different squares. These squares are: [... lists: d7, e8, g8, h7, d5, e4 ...] [...] _Me:_ You forgot g4 and h5, but no matter. Is d4 in that list? _ChatGPT:_ [...] To answer your other question, no, the square d4 is not one of the squares that a knight on f6 can move to directly. However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.
- kgwgk 3y ago
  > However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5. GPT stands for Goal Post Transportation.
- euroderf 3y ago
  So is this how hallucinations form ? It proposes something half-baked, and then later refers back to it as fact ?
  - abraxas 3y ago
    Sounds like we already have full blown cyber replicas of politicians then.
    - mythrwy 3y ago
      Or some bosses I've had. I actually can't wait to start reporting to GPT, how much crazier can it be?
  - 3y ago
jstanley 3y ago
I played chess against ChatGPT just yesterday, and it got into a winning position against me. After 24 moves it tried to play an illegal move and then when I told it that's illegal it played a bad move, and after that it didn't manage to find any more legal moves (I gave up after asking it to try again about 10 times). But it's very close to being able to play chess. My prompt was: > we're playing chess. only send chess moves. > > 1. e4
- zimpenfish 3y ago
  What does it do if you play illegal moves?
  - jstanley 3y ago
    Good idea, I tried that just now. It just accepts it and carries on.
    - TedDoesntTalk 3y ago
      It does the same when you ask it to be DM in a D&D game. It allows the players to do many, many things outside the rules. I don't remember any examples but a general idea was, "The character Frodo now has a the ability to breath fire. He breathes fire on the orcs." Although IIRC that was ChatGPT 3.5.
      - sebzim4500 3y ago
        I wonder if this would happen if you used the GPT-4 api with a system message. In my experience that can help but does necessarily solve the issue.
      - Closi 3y ago
        You can overcome this with prompt engineering. “If a player tries to do something not strictly within the rules of <insert game>, then you must inform me that it is an invalid move and not accept it” GPT appears to be slightly tuned to default to “yes and” in these ‘creative’ situations by default rather than “block/deny”. IMO lots of things where people don’t think GPT can do something end up being possible with basic prompt engineering. Usually people go too-short and too-non-specific on the prompt.
sangnoir 3y ago
I'm more worried about unethical applications of current ML SOTA than AGI. A trivial example would be removing humans out of the loop in antipersonnel armed drones so they autonomously classify targets and if/when to pull the trigger.
endtime 3y ago
> ChatGPT can't play chess, for example Do you really expect this claim to age well? I wouldn't be at all surprised if GPT5 could play chess quite well. The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it, but the worst possible time to start worrying about it is when we unquestionably have it. Because the whole point is that that is too late.
- pmoriarty 3y ago
  "The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it" He was far, far from the first. Warnings about the threat of intelligent machines have been around for over 100 years. The first instance I know of is E. M. Foster's "The Machine Stops"[1]. which was published in 1909, though I wouldn't be surprised if there were even earlier warnings. They've been well worn scifi tropes ever since. [1] - https://en.m.wikipedia.org/wiki/The_Machine_Stops https://en.m.wikipedia.org/wiki/The_Machine_Stops
  - ben_w 3y ago
    I'd say Frankenstein, and before that magical "didn't think it though" errors like The Sorcerer's Apprentice or Midas, or "didn't read the instructions" errors such as Grottasöngr[0] or 水母娘娘 and the magic whip[1]. That said, Yudkowsky seems (rightly or wrongly) to get the credit for turning this from merely fun stories into a serious (and not insane) field of study. [0] https://en.wikipedia.org/wiki/Grottasöngr https://en.wikipedia.org/wiki/Grottasöngr [1] https://en.wikipedia.org/wiki/Shuimu https://en.wikipedia.org/wiki/Shuimu
  - endtime 3y ago
    There's been a lot of sci-fi, I agree. I see that as quite distinct from Yudkowsky's efforts.
rst 3y ago
People have gotten it to play chess. The quality of play you get is very sensitive to prompt details, and may also be affected by OpenAI's ongoing model tuning, but here's one experiment in which it won 11 games of 19 against amateurs (treating any illegal move it tried to make as an immediate forfeit): https://dkb.blog/p/chatgpts-chess-elo-is-1400 https://dkb.blog/p/chatgpts-chess-elo-is-1400
HarHarVeryFunny 3y ago
It's not surprising that ChatGPT can't play chess well, since: 1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion. It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good. 2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much. 3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.

kranke155 3y ago

How would you curtail their use when any military that commits to using them will have a huge advantage ? This isn’t like nuclear weapons where any use is curtailed by the apocalyptic outcomes. Killer robots are the way we will fight in the future and any military which refuses to deploy them will find themselves facing defeat.

throw_nbvc1234 3y ago
AI is not allowed to make the decision to kill, that must be done by a human. Having every other step done by an AI is still going to be a huge advantage over non-AI military forces.
- michaelmior 3y ago
  Even if the decision is made by a human, it really matters how that decision is presented. My understanding of the current state of warfare (which could be way off) is that were possible, targets are still largely selected manually based on verified intelligence. What if we allow AI to propose targets? Even if testing shows that the AI is highly effective at picking "good" targets, that changes the situation entirely. Suppose humans make the final decision of which target to strike. How much time is given to make that decision? What information is presented to them? If the answer is that humans make a decision in a short amount of time largely relying on an AI-generated confidence score, are humans really making the decision?
  - Keegs 3y ago
    Exactly. It’s too easy for these decisions to become a formality. ProPublica’s reporting on Cigna[1] is a great example: > Before health insurers reject claims for medical reasons, company doctors must review them, according to insurance laws > A Cigna algorithm flags mismatches between diagnoses and what the company considers acceptable tests and procedures for those ailments Sounds fine, except > “We literally click and submit,” one former Cigna doctor said. “It takes all of 10 seconds to do 50 at a time.” [1]: https://www.propublica.org/article/cigna-pxdx-medical-health-insurance-rejection-claims https://www.propublica.org/article/cigna-pxdx-medical-health...

lumost 3y ago

There is such a blurry line for autonomous munitions. militaries used dumb imprecise munitions for decades - then precision weapons. A2A missiles used to lock on radar signature leading to huge risks related to accidentally shooting airliners/friendly craft. Now antiship missiles dynamically select their target over 300km away to maximize the chance of hitting a big ship. During the war on terror, ML models would decide which phone to blow up. We’re probably going to see ai driven target selection and prioritization for fire control within the next few months of the Ukraine war. The US’s new Rapid dragon program almost demands ai control of target selection and flight trajectories. Where do you draw the line? What would an appropriate agreement look like?

stareatgoats 3y ago
> Where do you draw the line? What would an appropriate agreement look like? Good questions. I'm not sure, but that line needs to be drawn somewhere. Thousands of experts should be pouring over questions like these, and likely will.
- JieJie 3y ago
  Here's an interview[0] with Lauren Khan from the Council on Foreign Relations[1] who is indeed pouring over those questions. [0]https://youtu.be/CDoIPpcwnCE https://youtu.be/CDoIPpcwnCE [1]https://www.cfr.org/expert/lauren-kahn https://www.cfr.org/expert/lauren-kahn
- milesward 3y ago
  Who pays for those experts, and what are their incentives?
neatze 3y ago
The line already drawn, no indiscriminate killings of civilians, all this technology only minimizes civilian suffering, furthermore it is more humane to use terminators instead of 20 years olds clearing buildings and autonomous tanks/tracks driving though IEDs filled streets. War is deeply in human nature, it is not going away, question is; how to make it more humane and minimize indiscriminate killings and sufferings in most cases poorest populations on earth.

1024core 3y ago

> The tried and tested method is international agreements. You really think actors like North Korea, Al Qaeda, etc. will adhere to International agreements?!?

flangola7 3y ago
Against the combined militaries of the rest of the planet? Yes.
waboremo 3y ago
Yes, they adhere to many international agreements already. You can look up North Korea's signed treaties if you're unsure. Ignoring the Al Qaeda part (or similars) because a fragile extremist group barely held together is unlikely to have the resources to assemble anything devastating enough to warrant infringing on an international agreement regarding militarized AI.
mtlmtlmtlmtl 3y ago
I'm not terribly worried about Al Qaeda or DPRK having any sort of capability to develop planetkilling AI, now or in the future. Nukes and Rockets? Sure. Anyone can build a centrifuge, and anyone can build a cylinder with explosives on it. But intelligent killer robots? Please.

slashdev 3y ago

It’s not the war robots that worry me as much as centralized intelligence with internet connectivity. War robots don’t reproduce, require energy infrastructure, and can be destroyed. While they could run amok, by targeting things they’re not supposed to, they won’t really be intelligent because the problem doesn’t require much intelligence. Now if they’re controlled by a central intelligence that’s a bit scarier.

Eisenstein 3y ago
What if the only way to stop war from occurring is to entirely remove humans from the decision making that would cause it?
- slashdev 3y ago
  You don’t get that with war robots, the decision to go to war or not is still in the hands of the politicians. But I wonder sometimes what it would take to end war.

uoaei 3y ago

I can't really tell if he's had a sincere change of heart about it. Certainly his screeds about how DL is the only path forward for AGI rang extremely hollow even 2 or 3 years ago. Those comments were clearly motivated by profit, considering his position in the field and all the companies vying for him at the time.

ecnahc515 3y ago

Let's hope we don't get to Horizon Zero Dawn too soon.

gumballindie 3y ago

An EMP bomb can easily sort out robots but nothing can protect us from data and ip theft. That’s the real danger here unless regulated quickly.

ryan93 3y ago

No one is uniquely positioned. Literally no one knows how powerful it will get.

O5vYtytb 3y ago

My biggest concern for military use of AI is how incompetent most military contractors are. These huge companies employ an army of not-very-good engineers whose primary purpose seems to be to over-complicate projects. Imagine the same teams that make planes which need to be hard rebooted every few days, now they're making advanced AI to dynamically target and kill people.

dmix 3y ago
> now they're making advanced AI to dynamically target and kill people. I'm curious, which projects are working on this? Any ones that we know about? In terms of production weapons I know the newer Switchblades, Excalibur, and even Javelin do some fancy automated targeting before the terminal phase but that's not really AI beyond image recognition and the targets/specific destination are pre-selected. I'm curious what sort of applications would use autonomous targeting without overt human involvement.
- ethbr0 3y ago
  Air defense has had automatic targeting since the 1980s(?) due to the required reaction times. As for autonomous target/kill programs, that'd be the USAF's Collaborative Combat Aircraft (CCA) program (under NGAD). It sounds like they're keeping human-in-the-loop for now by targeting a 1:many manned_stealth:drone ratio, but that will be subject to erosion as autonomous capabilities improve. And the very nature of the mission requires autonomy -- drones penetrating deep into sophisticated air defense systems using stealth and emissions control (so only line-of-sight communication links). Nobody is going to be remotely piloting these things.
- ikekkdcjkfke 3y ago
  You can bet they have automatic target tracking and ballistic aiming onto those in the new abrams tank, but sending the kill command will most likely be done by operators. But it's just a small step from that to putting it on autopilot
  - dmix 3y ago

nobodyandproud 3y ago

Outcome: Automate the economy, and employ the dispossessed to kill one another in the name of ethics (because AI military is unethical). This seems weird and arbitrary.

ren_engineer 3y ago

Military application of AI drones isn't even the worst possible use, it's nations using them to completely subjugate their own population(although the same tech could be used against non-peer nations). Combination of things like Gorgon Stare to direct smaller AI controlled drones like what they are using in Ukraine would be a police state nightmare. https://en.wikipedia.org/wiki/Gorgon_Stare https://en.wikipedia.org/wiki/Gorgon_Stare https://longreads.com/2019/06/21/nothing-kept-me-up-at-night-the-way-the-gorgon-stare-did/ https://longreads.com/2019/06/21/nothing-kept-me-up-at-night... they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with. No form of protest or movement against the government will be possible if it's scaled up

bluetwo 3y ago
At a music festival last summer I counted 4 drones in the sky monitoring 24/7 over 5 days. Never saw them come down. They were each tethered to a base station which seemed to run electric and probably returned a video feed. I expect to see this everywhere.
- senbrow 3y ago
  Which festival? And where?
- heavyset_go 3y ago
  The last few years has had the police flying drones over events where I am. Think like fairs, parades, etc.
l3mure 3y ago
Yup. [1] > Pentagon testing mass surveillance balloons across the US [2] > For instance, using Gorgon Stare, a 24-hour aerial surveillance system, the U.S. Air Force had been able to plot back from a car bomb explosion in Kabul in 2019, which killed 126 civilians, to find the location of safe houses used to execute the attack. [1] - https://www.theguardian.com/us-news/2019/aug/02/pentagon-balloons-surveillance-midwest https://www.theguardian.com/us-news/2019/aug/02/pentagon-bal... [2] - https://warontherocks.com/2023/04/ai-at-war/ https://warontherocks.com/2023/04/ai-at-war/

deskamess 3y ago

International treaties can hold to an extent. The greatest damage will be its internal use. Where countries can tell others to 'not interfere' in local business. Each country will run its own nefarious program and it will take a violent revolution to overthrow governments - and the next one will pick up the AI baton where the previous one left with a slogan of 'making sure no one does what the previous govt did'. So instead of an international global AI issue we will have strong national AI abuse. In either case, democracy will be put under strain.

dukeofdoom 3y ago

Leading theory is that COVID was made in a lab. Not sure what to fear more AI robots, or AI engineered viruses.

deleted 3y ago

[deleted]

heavyset_go 3y ago

Black Mirror got it right in their "Metalhead" episode, which is probably my favorite.

nradov 3y ago

International agreements are hardly tried and tested. The Nonproliferation Treaty has been somewhat effective with nuclear weapons largely because refining operations are hard to hide, and even with that several additional countries have acquired such weapons. Agreements on chemical and biological weapons are largely moot because it turns out that such weapons aren't even very effective compared to kinetic alternatives. The ban on land mines was never ratified by the countries that do most fighting, and such mines are being heavily used by both sides in Ukraine. The Washington Naval Treaty was a total failure. The ban on space weapons is breaking down right now. It is impossible to have an effective international agreement on autonomous weapons. No military power would ever agree to let a third party inspect their weapon source code in a verifiable way. It's too easy to hide the real code, and we would never trust potential adversaries not to cheat. Fully autonomous weapons have already been deployed for decades. The Mark 60 CAPTOR mine could sit and wait for weeks until it detected a probable target matching a programmed signature, then launch a homing torpedo at it. After the initial deployment there is no human in the loop.

stareatgoats 3y ago
FWIW, you make a good point, albeit a pessimistic one - at least if you like me dread the chicken race towards doom that is the obvious alternative. Agree that international agreements may not always have been the success stories I implied. But even the fact that some states have not signed certain treaties prove that they are effective (otherwise they could have signed and then just ignored them). Once a state has signed a treaty you won't need to rely on outside inspectors exclusively - inside whistleblowers will do just fine.
- nradov 3y ago
  How many inside whistleblowers do you think we will get from China, North Korea, and Iran?
  - stareatgoats 3y ago
    Whistleblowers, spies, side-channel intelligence. Not that many tools but enough to make most states avoid entering agreements they won't honor. No state wants to be exposed as a deceptive liar, it goes to their legitimacy. Impossible? well believe what you like. I'd like to believe that it is possible.

dan-robertson 3y ago

Sure, his position is reasonably unique, and he’s potentially had a broad overview of lots of things going on at Google and the industry in general, but is your claim that he is good at pointing out dangers because he hears lots of gossip, or is it that being involved in deep learning for a long time makes him good at figuring out those things. I definitely don’t buy the latter. What, precisely, is the reason you think Hinton would be good at pointing out dangers? Maybe you just mean that journalists will be happy to interview him rather than that he is likely to be right? Certainly that does give one an advantage in pointing things out.

api 3y ago

The scenario that I find both most scary and most likely is the use of AI to propagandize, brainwash, and con human beings at scale. Basically you can now assign every single living human being their own 24/7 con artist and power that con artist with reams of personalized surveillance information about each target purchased from data brokers. Everyone will have a highly informed personalized con artist following them around 24/7 trying to convince them of whatever the controller of that bot has programmed it to sell. We're creating the propaganda equivalent of the hydrogen bomb.

ericmcer 3y ago

Threats like this seem less real to me because the government has been so technologically inept lately. Garbage government websites, failed rollouts of huge programs (like healthcare, the CA highspeed rail), SpaceX taking the reigns away from NASA and the military awarding giant contracts to Amazon and Microsoft to keep their ancient tech infra running. It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them.

DrBenCarson 3y ago
Um yeah...someone sells the government all of its weapons. Literally all of them. It would be a dramatic change for the government to in-source weapons development.
alecbz 3y ago
I don't know too much about the internals of the military, but I think they already have a lot of very impressive technology that would seem at-odds with other signs of governmental ineptitude with technology. A government website being shit might not be viewed with as much urgency and importance as combat technology. > It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them. I mean, that might be how it happens? Military contractors might as well be seen as a part of the military.