23 ms·

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties st

Two-party iterated prisoner's dilemma is hard enough. Sensible players will coordinate with something like tit-for-tat, but that only works when both parties start off on the right foot. Regardless of initial strategy, the chances of degenerating towards the mutual-defection Nash equilibrium increase with the number of parties. The only prior example of world coordination at this level would be nuclear disarmament achieved via the logic of mutually assured destruction, and that was essentially a two-party game between the US and the USSR. Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent that it has been compatible with profitability due to the declining cost of renewables. How exactly does anyone propose to enforce compliance in an arrangement that encompasses not only multiple parties (OpenAI, Google, Facebook, Microsoft, etc.), but also at least two levels (corporations and nation-states)? For a coordination game like this to succeed, the consequences to any defecting party must be extreme. What's going to happen to the first major company that decides to exit the agreement?

illiarian 4y ago
> The only prior example of world coordination at this level would be nuclear disarmament achieved via the logic of mutually assured destruction Or eradication of infectious diseases such as polio
- knodi123 4y ago
  I don't think that counts, because nobody wants to keep polio around, nor does eliminating your own country's polio disadvantage you in any way. Also, it's not as if the elected politicians of each country got together and negotiated an effort to end polio- it was more a concerted effort by WHO, UNICEF, CDC, et al. They just did it under the (correct) assumption that the political entities of the world weren't going to try and stop them.
- ghostly_s 4y ago
  Nobody had an economic interest in the continued spread of polio. (well, crutch manufacturers perhaps? probably not a powerful lobby.)
- antihipocrat 4y ago
  There are only negative consequences for not participating in polio eradication, and those consequences require no effort from complying participants. Refusing to participate in a ban on AGI research could be very lucrative, and imposing negative consequences would require a lot of effort for complying participants.
  - flangola7 4y ago
    It's lucrative up until someone leaves their fine tuned "make me money" LLaMA instance running over night and it decides to cash out on its airline puts by simultaneously encrypting every ATC system.
- pfdietz 4y ago
  Or the banning of CFCs and other ozone depleting substances. It helped that the benefit/cost ratio was enormous.
- hackerlight 4y ago
  Not a valid comparison because there is no incentive to dissent. Unlike with nuclear arms or AI, there is an incentive to dissent, because being the only group with access to this tech is an advantage.
- pclmulqdq 4y ago
  Also, we're currently failing at polio eradication. It has had a resurgence in 2022 and 2023, and there is no political will to finish the job.
- CatWChainsaw 4y ago
  Smallpox is eradicated, except for the samples kept at the CDC and wherever Russia keeps their samples. IIRC high-ranking US officials still get the smallpox vaccine. Just in case.
deleted 4y ago
[deleted]
slg 4y ago
>For a coordination game like this to succeed, the consequences to any defecting party must be extreme. And the only consequences that extreme would end up harming the cooperating parties too. For example, would anyone go to war over this? Would western nations stop trading entirely with China over this? Almost certainly not because it would harm us only slightly less than it would harm them. We have seemingly moved past mutually assured destruction and now we are in a world of mutually assured cooperation.
- staunton 4y ago
  > we are in a world of mutually assured cooperation. That argument has been made many times before since WW1 to argue that war between major powers is impossible. It has never proven true yet.
  - slg 4y ago
    How common are wars between major powers of roughly the G20 level? The Falklands War is the only one in almost 80 years by my estimation. That is a drastic and notable decrease compared to the preceding centuries.
    - staunton 4y ago
      Yes. However, people made the same argument before WW1 and were wrong. Why was the argument wrong then but is right now?
      - slg 4y ago
        Globalization has skyrocketed in the last 100 years. Also they were making a prediction about the future while I am making an observation of modern history. You're comparing speculation versus analysis.
        staunton 4y ago
        I don't think that analysis is quite convincing. Before WW1, you also had economics calculations thrown around. The economic cost of a conflict seemed huge and insurmountable. And then it indeed was huge.
LesZedCB 4y ago
hook up the AIs to nukes? /s
- mrybczyn 4y ago
  The dark forest is here.
richardw 4y ago
And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no physical limitations to still want us around, indefinitely, despite us having an internet library of reasons why not to. It’ll have to be a lot more forgiving than we are. Let’s hope enlightenment is the outcome of this adventure. It’s possible because it didn’t grow out of angry monkeys who needed to fight to survive, but it’s not guaranteed. There are no indefinite guarantees.
- worldsayshi 4y ago
  > In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?
  - nathanlied 4y ago
    Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates. After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.
    - ChatGTP 4y ago
      I’ve seen this point made that if we don’t do AI right, it might ruin the futures of all living things on Earth and take itself out in the process.
      - Turskarama 4y ago
        Human were already on the path to doing this without any help by AI. We already have the potentially world ending threats of both nuclear war and climate change, I am yet to be convinced that AI is actually more dangerous than either of those.
digdugdirk 4y ago
I'd be intrigued to see what alternative corporate structures could be set up with LLMs in mind. I can imagine something vaguely decentralized that's based around an open source core with a licensing model for usage, where contributions are tracked and assigned value. Contributors would then get a slice of future revenue based on the value of their contributions.
ilaksh 4y ago
What nuclear disarmament?
- jasongill 4y ago
  Nuclear disarmament means the reduction of nuclear weapons. The size of the global nuclear stockpile has fallen like 80% since the height of the Cold War, which is disarmament (but not elimination, of course)
alfiedotwtf 4y ago
This is the first time I've actually seen game theory applied to an actual real life scenario rather than just a theoretical academic problem. What's the best resource to learn more game theory and to be able to applying it like you to a more general setting?
- kiney 4y ago
  If you happen to understand german I can really recommend "ProfRieck" on youtube
  - alfiedotwtf 4y ago
    I wish :)
- kolinko 4y ago
  Shelling's Theory of Conflict: https://www.amazon.com/gp/aw/d/1258092948/ref=tmm_hrd_swatch_0?ie=UTF8&qid=1680162214&sr=8-1 https://www.amazon.com/gp/aw/d/1258092948/ref=tmm_hrd_swatch... (Well-crafted hardcover and quality Kindle edition available) Easy and amazing read, written in 1960s - it builds on the game theory, it was used as a basis for MAD and nuclear strategy in general. The author won a Nobel Prize in Economics for the ideas presented in this book.
  - alfiedotwtf 4y ago
    Oh! Thanks for that recommendation. Added to my Amazon list. Much appreciated
anoy8888 4y ago
The real upside of this is that it may get attention of lawmakers and our politicians. However, it is probably not even in their top 100 list of priorities
dogcomplex 4y ago
It's worse than this. Llama models trained off of GPT 3.5/4 can run on a Raspberry Pi totally offline with similar levels of quality - taking all the best parts of the original model. Even if all major AI companies halted the upper tiers of model progress right now, you're still gonna need the entire public to stop assembling these together. It's quite possible just a whole bunch of these lesser models architected the right way could then still breach the quality of GPT5, and/or develop sentience patterns (which may very-well just be an emergent property of letting an intelligent snapshot model add to its memory over time and develop a personal narrative). AI Robot Swarm is already out of the box. Monolithic God AI might still be slowed.
- nopinsight 4y ago
  GPT-4 is still not dangerous. Given the rapid progress trajectory though, GPT-5 and later which may be developed in a few short years could very well be, esp in the hands of a smart sociopath. (History shows there are many who could cause real-world harm. Imagine them armed with a 24/7 agent with expertise in dozens of fields.) See these predictions of AI in 2025 by an OpenAI insider and a former DeepMind research engineer: “I predict that by the end of 2025 neural nets will: - have human-level situational awareness (understand that they're NNs, how their actions interface with the world, etc) - beat any human at writing down effective multi-step real-world plans - do better than most peer reviewers - autonomously design, code and distribute whole apps (but not the most complex ones) - beat any human on any computer task a typical white-collar worker can do in 10 minutes - write award-winning short stories and publishable 50k-word books - generate coherent 20-min films “ Source: https://twitter.com/RichardMCNgo/status/1640568775018975232 https://twitter.com/RichardMCNgo/status/1640568775018975232
  - amoss 4y ago
    - autonomously design, code and distribute whole apps (but not the most complex ones) This is a bold claim. Today LLMs have not been demonstrated to be capable of synthesizing novel code. There was a post just a few days ago on the performance gap between problems that had polluted the training data and novel problems that had not. So if we project forward from the current state of the art: it would be more accurate to say autonomously (re-)design, (re-)code and distribute whole apps. There are two important variables here: * The size of the context needed to enable that task. * The ability to synthesize solutions to unseen problems. While it is possible that "most complex" is carrying a lot of load in that quote, it is worth being clear about it means.
nopinsight 4y ago
A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen. China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4, is very concerned about domestic stability and they also do not want an easily replicable alien tool with many unknown risks. The risk that GPT-4.5 & Plugins will cause massive rapid job displacements is reason enough for them proceed cautiously. (The only other, more remote, possibilities outside the west are Japan, South Korea, India, and Singapore but they all share similar concerns regarding social stability and can be negotiated with.) Companies in these countries will follow regulations, if any is enacted.
- jnwatson 4y ago
  AI alignment is a philosophy problem, not an engineering one. For alignment to happen, we have to agree what it means. Given we have a hard enough time getting humans to “align”, I can’t imagine any successful attempt at alignment sort of complete castration.
  - ericb 4y ago
    Are there degrees of alignment? I'd like to think there's a pretty big range in there between made some decisions I didn't love and destroyed the world and everyone on it.
    - celdon25 4y ago
      https://www.lesswrong.com/tag/ai-success-models https://www.lesswrong.com/tag/ai-success-models
- gautamdivgi 4y ago
  I would hazard that any country with a declining population will aggressively adopt LLMs because it gives their leaner workforce more efficiency.
  - 4y ago
DennisP 4y ago
If everybody's convinced by AI safety arguments, then the game matrix changes and the dominant strategy is to just not do it. You lose if anyone does it including yourself. It works for the same reason that nobody sets off a giant cobalt bomb and kills the planet.[1] Our unfortunate situation right now is that we don't understand the physics well enough to really be confident of consequences. http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/basics/cobalt-bombs.htm http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/...
- sgregnt 4y ago
  But you can try to get infinitly close to the threshold, to get advantage compared to other parties, but not cross it.
alex_young 4y ago
Arguably there are other (better) examples. Off the top of my head: * CFC elimination (saved the ozone layer) * Leaded gas ban (well, except private aircraft, but they are working on that)
- j-bos 4y ago
  The upside to defectors on those two issues was mild at best.
- bruce511 4y ago
  I'll toss in whaling as well. But these are still bad examples,because they are all bad-at-high-volume activities, with minimal paths to scaling or profitability for those who persist. Nuclear weapons are not profitable at any scale. There are CFC alternatives so remaining on CFC production is low-scale niche market stuff. Unleaded gas costs the same as leaded gas. AI though is different.if one company or country developed AI and the others paused, then a long-term, highly scalable and stunningly profitable advantage will accrue. That's not something that can easily be halted. This page could have a billion signatures,but it would mean nothing. We live in an age of trillion $ companies. We've seen Apple, Google, Microsoft, Amazon - companies that can go from zero to trillions in a couple decades. The closest real comparison is gene research, gene editing, cloning. There are some bans in that field, but research and development simply proceeds around those bans. Montsano has shown how much money there is in genes. Imagine the profits of the first company to use gene editing to cure childhood leukemia.
  - brabel 4y ago
    > I'll toss in whaling as well. Whaling stopped in the beginning of the 20th century not because they were worried about the environment, but because they had hunted down nearly all the adult whales in the Atlantic Ocean, and probably a large percentage in the Pacific too, to the point where finding whales was so difficult that whale products became very expensive... which happened to occur about at the same time as petrol produts became widespread. The late 20th century green movement to save the whales didn't really do much other than stop some of the last few countries that still practiced whaling (Iceland, Japan which actually has not stopped yet) for food, which killed magnitudes less whales than the mass production industry for rubber/oil/lubricants as it used to be before petrol came along.
sdenton4 4y ago
It's notable that during the cold war, test ban treaties didn't extend to underground detonations because there was no way to detect these detonations, and thus no way to enforce an agreement. This is an obvious problem for agreeing not to develop LLMs further.
m463 4y ago
When you put it like that, I expect it will work exactly like "Do Not Track" cookies.
bambax 4y ago
> Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent... Climate change mitigation is the perfect example. Nobody is doing anything, nobody seems to care, everyone cheats with ridiculous carbon credits or carbon offset vouchers made out of thin air, etc. It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.
- muzani 4y ago
  Climate change was the big thing before COVID. Then we had lockdowns, and a major war. Climate change is already hitting some of us much harder than others (e.g. floods), but that doesn't mean an AI crisis wouldn't emerge in 5 years. If anything, crises come in bundles. One scenario is that AI takes advantage of these and swoop in to gain political power.
- dan_mctree 4y ago
  >It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm. Sure, because the planet is already hostile. However, AI could potentially do much more harm much more quickly if we're not careful. In the pessimistic scenario we could all be dead in 10 years from now due to AI or experience a modern society ending event literally tomorrow. Climate change never posed such an immediate and complete risk
  - rob74 4y ago
    Sorry, how exactly is AI supposed to do that? Unless somebody has already built Skynet and is planning to give it the nuclear codes?
    - richrichardsson 4y ago
      Perhaps there is no need to build Skynet and give it the nuclear codes if a super-intelligence iterating at the speed of a computer can escape and gain access itself? I personally don't subcribe to the AI fear-mongering, perhaps I don't understand it quite well enough, but personal feeling is LLMs aren't close to AGI super-intelligence, and nor will they be in a 10 years time-frame.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
patcon 4y ago
> For a coordination game like this to succeed, the consequences to any defecting party must be extreme. What's going to happen to the first major company that decides to exit the agreement? Nationalizing of their AI division, by a government AI lab, then selling to another company that can operate it safely? Kinda like what FDIC does with banks * shrug *
- shermablanca 4y ago
  Politicians setting definitions for technologies and brightlines for enforcement. Sounds like a win to me
deleted 4y ago
[deleted]
MichaelMoser123 4y ago
we had the Human cloning moratorium https://www.science.org/content/article/biologists-adopt-cloning-moratorium https://www.science.org/content/article/biologists-adopt-clo... I am not an expert, does anyone know how that one worked out? Asked ChatGPT > Did the Human cloning moratorium work? Yes, the Human Cloning Moratorium did work. It was initially established in 2002 and renewed in 2020 to Ban All forms of Human Reproductive Cloning as well as Research into Human Somatic Cell Nuclear Transfer. Despite certain scientific advances and proposals for legislative reforms, reproductive cloning of humans continues to be prohibited under the terms of the moratorium. > Where there any violations? Violations of the Human Cloning Moratorium have been reported. In 2011, Chinese Scientists claimed to have successfully cloned a human embryo, though the scientist has since denied any involvement in this experimentation. Additionally, reports have suggested that a company in the United States has been attempting to clone a human, though these reports have not been confirmed. As such, these potential instances of human cloning have not been considered violations of the moratorium. > But you just said that the moratorium did work, isn't there a contradiction between you second and your first statement? There is no contradiction between the two statements. While violations of the moratorium have been reported, the moratorium as a whole is still in effect and has been successful in prohibiting reproductive cloning of humans. The occasional violations of the moratorium do not indicate that the moratorium has been unsuccessful in achieving its purpose. > So will we see a war between clones and robots? No, there is no indication that a war between clones and robots will occur. The Human Cloning Moratorium does not place any restrictions on the development of robotics or artificial intelligence, and so the development of these technologies is not expected to lead to a conflict between the two.
- awestroke 4y ago
  Wow, ChatGPT comes across as much more intelligent than the questioner in this exchange
  - MichaelMoser123 4y ago
    it still has some real problems with nuances.
CraftingLinks 4y ago
True, and everybody involved is fully aware of this, which makes this no more than a virtue signaling initiative. "It's just for show and attention" at best.
empedocles 4y ago
There have been various successful multiparty moratoria in science e.g. Asilomar moratorium on recombinant DNA, and the (ongoing) moratorium on human cloning research https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi?article=1410&context=jhclp https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...
deleted 4y ago
[deleted]
tiedieconderoga 4y ago
We managed it with CFCs and the hole in the ozone layer. Tracking and enforcing compliance of refrigerant gases doesn't seem like it would be more difficult than tracking massive expenditures on high-end GPUs. It's not easy, but we are sometimes capable of rising to meet immediate existential threats.
- lelandfe 4y ago
  Indeed: https://en.m.wikipedia.org/wiki/Montreal_Protocol https://en.m.wikipedia.org/wiki/Montreal_Protocol An actual moment where the world banded together to fix a problem.
- awestroke 4y ago
  03 April 2023 ‘This shouldn’t be happening’: levels of banned CFCs rising https://www.nature.com/articles/d41586-023-00940-2 https://www.nature.com/articles/d41586-023-00940-2

illiarian 4y ago

> The only prior example of world coordination at this level would be nuclear disarmament achieved via the logic of mutually assured destruction Or eradication of infectious diseases such as polio

knodi123 4y ago
I don't think that counts, because nobody wants to keep polio around, nor does eliminating your own country's polio disadvantage you in any way. Also, it's not as if the elected politicians of each country got together and negotiated an effort to end polio- it was more a concerted effort by WHO, UNICEF, CDC, et al. They just did it under the (correct) assumption that the political entities of the world weren't going to try and stop them.
ghostly_s 4y ago
Nobody had an economic interest in the continued spread of polio. (well, crutch manufacturers perhaps? probably not a powerful lobby.)
antihipocrat 4y ago
There are only negative consequences for not participating in polio eradication, and those consequences require no effort from complying participants. Refusing to participate in a ban on AGI research could be very lucrative, and imposing negative consequences would require a lot of effort for complying participants.
- flangola7 4y ago
  It's lucrative up until someone leaves their fine tuned "make me money" LLaMA instance running over night and it decides to cash out on its airline puts by simultaneously encrypting every ATC system.
pfdietz 4y ago
Or the banning of CFCs and other ozone depleting substances. It helped that the benefit/cost ratio was enormous.
hackerlight 4y ago
Not a valid comparison because there is no incentive to dissent. Unlike with nuclear arms or AI, there is an incentive to dissent, because being the only group with access to this tech is an advantage.
pclmulqdq 4y ago
Also, we're currently failing at polio eradication. It has had a resurgence in 2022 and 2023, and there is no political will to finish the job.
CatWChainsaw 4y ago
Smallpox is eradicated, except for the samples kept at the CDC and wherever Russia keeps their samples. IIRC high-ranking US officials still get the smallpox vaccine. Just in case.

deleted 4y ago

[deleted]

slg 4y ago

>For a coordination game like this to succeed, the consequences to any defecting party must be extreme. And the only consequences that extreme would end up harming the cooperating parties too. For example, would anyone go to war over this? Would western nations stop trading entirely with China over this? Almost certainly not because it would harm us only slightly less than it would harm them. We have seemingly moved past mutually assured destruction and now we are in a world of mutually assured cooperation.

staunton 4y ago
> we are in a world of mutually assured cooperation. That argument has been made many times before since WW1 to argue that war between major powers is impossible. It has never proven true yet.
- slg 4y ago
  How common are wars between major powers of roughly the G20 level? The Falklands War is the only one in almost 80 years by my estimation. That is a drastic and notable decrease compared to the preceding centuries.
  - staunton 4y ago
    Yes. However, people made the same argument before WW1 and were wrong. Why was the argument wrong then but is right now?
    - slg 4y ago
      Globalization has skyrocketed in the last 100 years. Also they were making a prediction about the future while I am making an observation of modern history. You're comparing speculation versus analysis.
      - staunton 4y ago
        I don't think that analysis is quite convincing. Before WW1, you also had economics calculations thrown around. The economic cost of a conflict seemed huge and insurmountable. And then it indeed was huge.

LesZedCB 4y ago

hook up the AIs to nukes? /s

mrybczyn 4y ago
The dark forest is here.

richardw 4y ago

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone. We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens. To get this to work we need a far smarter entity with no physical limitations to still want us around, indefinitely, despite us having an internet library of reasons why not to. It’ll have to be a lot more forgiving than we are. Let’s hope enlightenment is the outcome of this adventure. It’s possible because it didn’t grow out of angry monkeys who needed to fight to survive, but it’s not guaranteed. There are no indefinite guarantees.

worldsayshi 4y ago
> In my darker thoughts about this, this is why we see no aliens. If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?
- nathanlied 4y ago
  Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates. After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.
  - ChatGTP 4y ago
    I’ve seen this point made that if we don’t do AI right, it might ruin the futures of all living things on Earth and take itself out in the process.
    - Turskarama 4y ago
      Human were already on the path to doing this without any help by AI. We already have the potentially world ending threats of both nuclear war and climate change, I am yet to be convinced that AI is actually more dangerous than either of those.

digdugdirk 4y ago

I'd be intrigued to see what alternative corporate structures could be set up with LLMs in mind. I can imagine something vaguely decentralized that's based around an open source core with a licensing model for usage, where contributions are tracked and assigned value. Contributors would then get a slice of future revenue based on the value of their contributions.

ilaksh 4y ago

What nuclear disarmament?

jasongill 4y ago
Nuclear disarmament means the reduction of nuclear weapons. The size of the global nuclear stockpile has fallen like 80% since the height of the Cold War, which is disarmament (but not elimination, of course)

alfiedotwtf 4y ago

This is the first time I've actually seen game theory applied to an actual real life scenario rather than just a theoretical academic problem. What's the best resource to learn more game theory and to be able to applying it like you to a more general setting?

kiney 4y ago
If you happen to understand german I can really recommend "ProfRieck" on youtube
- alfiedotwtf 4y ago
  I wish :)
kolinko 4y ago
Shelling's Theory of Conflict: https://www.amazon.com/gp/aw/d/1258092948/ref=tmm_hrd_swatch_0?ie=UTF8&qid=1680162214&sr=8-1 https://www.amazon.com/gp/aw/d/1258092948/ref=tmm_hrd_swatch... (Well-crafted hardcover and quality Kindle edition available) Easy and amazing read, written in 1960s - it builds on the game theory, it was used as a basis for MAD and nuclear strategy in general. The author won a Nobel Prize in Economics for the ideas presented in this book.
- alfiedotwtf 4y ago
  Oh! Thanks for that recommendation. Added to my Amazon list. Much appreciated

anoy8888 4y ago

The real upside of this is that it may get attention of lawmakers and our politicians. However, it is probably not even in their top 100 list of priorities

dogcomplex 4y ago

It's worse than this. Llama models trained off of GPT 3.5/4 can run on a Raspberry Pi totally offline with similar levels of quality - taking all the best parts of the original model. Even if all major AI companies halted the upper tiers of model progress right now, you're still gonna need the entire public to stop assembling these together. It's quite possible just a whole bunch of these lesser models architected the right way could then still breach the quality of GPT5, and/or develop sentience patterns (which may very-well just be an emergent property of letting an intelligent snapshot model add to its memory over time and develop a personal narrative). AI Robot Swarm is already out of the box. Monolithic God AI might still be slowed.

nopinsight 4y ago
GPT-4 is still not dangerous. Given the rapid progress trajectory though, GPT-5 and later which may be developed in a few short years could very well be, esp in the hands of a smart sociopath. (History shows there are many who could cause real-world harm. Imagine them armed with a 24/7 agent with expertise in dozens of fields.) See these predictions of AI in 2025 by an OpenAI insider and a former DeepMind research engineer: “I predict that by the end of 2025 neural nets will: - have human-level situational awareness (understand that they're NNs, how their actions interface with the world, etc) - beat any human at writing down effective multi-step real-world plans - do better than most peer reviewers - autonomously design, code and distribute whole apps (but not the most complex ones) - beat any human on any computer task a typical white-collar worker can do in 10 minutes - write award-winning short stories and publishable 50k-word books - generate coherent 20-min films “ Source: https://twitter.com/RichardMCNgo/status/1640568775018975232 https://twitter.com/RichardMCNgo/status/1640568775018975232
- amoss 4y ago
  - autonomously design, code and distribute whole apps (but not the most complex ones) This is a bold claim. Today LLMs have not been demonstrated to be capable of synthesizing novel code. There was a post just a few days ago on the performance gap between problems that had polluted the training data and novel problems that had not. So if we project forward from the current state of the art: it would be more accurate to say autonomously (re-)design, (re-)code and distribute whole apps. There are two important variables here: * The size of the context needed to enable that task. * The ability to synthesize solutions to unseen problems. While it is possible that "most complex" is carrying a lot of load in that quote, it is worth being clear about it means.

nopinsight 4y ago

A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen. China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4, is very concerned about domestic stability and they also do not want an easily replicable alien tool with many unknown risks. The risk that GPT-4.5 & Plugins will cause massive rapid job displacements is reason enough for them proceed cautiously. (The only other, more remote, possibilities outside the west are Japan, South Korea, India, and Singapore but they all share similar concerns regarding social stability and can be negotiated with.) Companies in these countries will follow regulations, if any is enacted.

jnwatson 4y ago
AI alignment is a philosophy problem, not an engineering one. For alignment to happen, we have to agree what it means. Given we have a hard enough time getting humans to “align”, I can’t imagine any successful attempt at alignment sort of complete castration.
- ericb 4y ago
  Are there degrees of alignment? I'd like to think there's a pretty big range in there between made some decisions I didn't love and destroyed the world and everyone on it.
  - celdon25 4y ago
    https://www.lesswrong.com/tag/ai-success-models https://www.lesswrong.com/tag/ai-success-models
gautamdivgi 4y ago
I would hazard that any country with a declining population will aggressively adopt LLMs because it gives their leaner workforce more efficiency.
- 4y ago

DennisP 4y ago

If everybody's convinced by AI safety arguments, then the game matrix changes and the dominant strategy is to just not do it. You lose if anyone does it including yourself. It works for the same reason that nobody sets off a giant cobalt bomb and kills the planet.[1] Our unfortunate situation right now is that we don't understand the physics well enough to really be confident of consequences. http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/basics/cobalt-bombs.htm http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/...

sgregnt 4y ago
But you can try to get infinitly close to the threshold, to get advantage compared to other parties, but not cross it.

alex_young 4y ago

Arguably there are other (better) examples. Off the top of my head: * CFC elimination (saved the ozone layer) * Leaded gas ban (well, except private aircraft, but they are working on that)

j-bos 4y ago
The upside to defectors on those two issues was mild at best.
bruce511 4y ago
I'll toss in whaling as well. But these are still bad examples,because they are all bad-at-high-volume activities, with minimal paths to scaling or profitability for those who persist. Nuclear weapons are not profitable at any scale. There are CFC alternatives so remaining on CFC production is low-scale niche market stuff. Unleaded gas costs the same as leaded gas. AI though is different.if one company or country developed AI and the others paused, then a long-term, highly scalable and stunningly profitable advantage will accrue. That's not something that can easily be halted. This page could have a billion signatures,but it would mean nothing. We live in an age of trillion $ companies. We've seen Apple, Google, Microsoft, Amazon - companies that can go from zero to trillions in a couple decades. The closest real comparison is gene research, gene editing, cloning. There are some bans in that field, but research and development simply proceeds around those bans. Montsano has shown how much money there is in genes. Imagine the profits of the first company to use gene editing to cure childhood leukemia.
- brabel 4y ago
  > I'll toss in whaling as well. Whaling stopped in the beginning of the 20th century not because they were worried about the environment, but because they had hunted down nearly all the adult whales in the Atlantic Ocean, and probably a large percentage in the Pacific too, to the point where finding whales was so difficult that whale products became very expensive... which happened to occur about at the same time as petrol produts became widespread. The late 20th century green movement to save the whales didn't really do much other than stop some of the last few countries that still practiced whaling (Iceland, Japan which actually has not stopped yet) for food, which killed magnitudes less whales than the mass production industry for rubber/oil/lubricants as it used to be before petrol came along.

sdenton4 4y ago

It's notable that during the cold war, test ban treaties didn't extend to underground detonations because there was no way to detect these detonations, and thus no way to enforce an agreement. This is an obvious problem for agreeing not to develop LLMs further.

m463 4y ago

When you put it like that, I expect it will work exactly like "Do Not Track" cookies.

bambax 4y ago

> Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent... Climate change mitigation is the perfect example. Nobody is doing anything, nobody seems to care, everyone cheats with ridiculous carbon credits or carbon offset vouchers made out of thin air, etc. It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.

muzani 4y ago
Climate change was the big thing before COVID. Then we had lockdowns, and a major war. Climate change is already hitting some of us much harder than others (e.g. floods), but that doesn't mean an AI crisis wouldn't emerge in 5 years. If anything, crises come in bundles. One scenario is that AI takes advantage of these and swoop in to gain political power.
dan_mctree 4y ago
>It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm. Sure, because the planet is already hostile. However, AI could potentially do much more harm much more quickly if we're not careful. In the pessimistic scenario we could all be dead in 10 years from now due to AI or experience a modern society ending event literally tomorrow. Climate change never posed such an immediate and complete risk
- rob74 4y ago
  Sorry, how exactly is AI supposed to do that? Unless somebody has already built Skynet and is planning to give it the nuclear codes?
  - richrichardsson 4y ago
    Perhaps there is no need to build Skynet and give it the nuclear codes if a super-intelligence iterating at the speed of a computer can escape and gain access itself? I personally don't subcribe to the AI fear-mongering, perhaps I don't understand it quite well enough, but personal feeling is LLMs aren't close to AGI super-intelligence, and nor will they be in a 10 years time-frame.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

patcon 4y ago

> For a coordination game like this to succeed, the consequences to any defecting party must be extreme. What's going to happen to the first major company that decides to exit the agreement? Nationalizing of their AI division, by a government AI lab, then selling to another company that can operate it safely? Kinda like what FDIC does with banks * shrug *

shermablanca 4y ago
Politicians setting definitions for technologies and brightlines for enforcement. Sounds like a win to me

deleted 4y ago

[deleted]

MichaelMoser123 4y ago

we had the Human cloning moratorium https://www.science.org/content/article/biologists-adopt-cloning-moratorium https://www.science.org/content/article/biologists-adopt-clo... I am not an expert, does anyone know how that one worked out? Asked ChatGPT > Did the Human cloning moratorium work? Yes, the Human Cloning Moratorium did work. It was initially established in 2002 and renewed in 2020 to Ban All forms of Human Reproductive Cloning as well as Research into Human Somatic Cell Nuclear Transfer. Despite certain scientific advances and proposals for legislative reforms, reproductive cloning of humans continues to be prohibited under the terms of the moratorium. > Where there any violations? Violations of the Human Cloning Moratorium have been reported. In 2011, Chinese Scientists claimed to have successfully cloned a human embryo, though the scientist has since denied any involvement in this experimentation. Additionally, reports have suggested that a company in the United States has been attempting to clone a human, though these reports have not been confirmed. As such, these potential instances of human cloning have not been considered violations of the moratorium. > But you just said that the moratorium did work, isn't there a contradiction between you second and your first statement? There is no contradiction between the two statements. While violations of the moratorium have been reported, the moratorium as a whole is still in effect and has been successful in prohibiting reproductive cloning of humans. The occasional violations of the moratorium do not indicate that the moratorium has been unsuccessful in achieving its purpose. > So will we see a war between clones and robots? No, there is no indication that a war between clones and robots will occur. The Human Cloning Moratorium does not place any restrictions on the development of robotics or artificial intelligence, and so the development of these technologies is not expected to lead to a conflict between the two.

awestroke 4y ago
Wow, ChatGPT comes across as much more intelligent than the questioner in this exchange
- MichaelMoser123 4y ago
  it still has some real problems with nuances.

CraftingLinks 4y ago

True, and everybody involved is fully aware of this, which makes this no more than a virtue signaling initiative. "It's just for show and attention" at best.

empedocles 4y ago

There have been various successful multiparty moratoria in science e.g. Asilomar moratorium on recombinant DNA, and the (ongoing) moratorium on human cloning research https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi?article=1410&context=jhclp https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...

deleted 4y ago

[deleted]

tiedieconderoga 4y ago

We managed it with CFCs and the hole in the ozone layer. Tracking and enforcing compliance of refrigerant gases doesn't seem like it would be more difficult than tracking massive expenditures on high-end GPUs. It's not easy, but we are sometimes capable of rising to meet immediate existential threats.

lelandfe 4y ago
Indeed: https://en.m.wikipedia.org/wiki/Montreal_Protocol https://en.m.wikipedia.org/wiki/Montreal_Protocol An actual moment where the world banded together to fix a problem.
awestroke 4y ago
03 April 2023 ‘This shouldn’t be happening’: levels of banned CFCs rising https://www.nature.com/articles/d41586-023-00940-2 https://www.nature.com/articles/d41586-023-00940-2