4 ms·

I'm not a historian, but I think that if in 1900, when there were no successful airplanes, if you tried to have serious dialogues on how to design safe air traf

by abeppu 4y ago

I'm not a historian, but I think that if in 1900, when there were no successful airplanes, if you tried to have serious dialogues on how to design safe air traffic control systems, you would have been laughed at. Maybe you'd be able to anticipate some real problems, but some of your focus would be wildly off. And to the extent that you could see real issues coming, it would be heavily informed by what you could deduce about planes and how they would be used. And those skills would probably have been better directed at making trains or cars safer. So I'm always a little surprised how seriously people take AI safety and alignment concerns about systems we don't yet know how to build, and how uninterested many of those same people are about 'alignment' issues with organizations and institutions we have today. And given our poor success in addressing our current issues, even where good policy proposals exist, why should we believe that technical solutions for AI alignment would be used even if they're researched and well-described?

ditchfieldcaleb 4y ago
You're right - the odds of success are incredibly slim. But we may only have one shot, because the first mid-air collision in this analogy could literally be the end of the world.
- faddypaddy34 4y ago
  How exactly? Explain how an AI would cause the end of the world? Are you suggesting we would turn over all of the world's nuclear arsenals to AI to deal with? Maybe it's just me but lately it seems like everything is being labeled as "dangerous" to the point of absurdity. It seems to be following the same line as US political rhetoric where everyone is either a Nazi or communist bent on destroying the country depending on what side you generally align with.
  - edanm 4y ago
    Forget the idea of an "AI" then, because the idea of "intelligence" makes the argument harder. Just think of a "new technology." Is it possible that a new technology could destroy the world? Of course. It could've turned out that nuclear weapons would incinerate the atmosphere upon detonation, as some were worried they would. It could be that the next technological innovation will kill us, there's nothing prevent it in the laws of physics. AGI is a specific technology we are worried about, because the whole premise is "once we build something that is extremely capable at a variety of things, one thing it will be capable of is destroying the world. Even by accident." We're already using AI techniques to help with problems in biology like protein folding. Take it a few dozen iterations forward, and these systems will be helping design medicines and vaccines that no human can do by themselves. At that point, what's to stop the system from creating a super-flu that kills everyone? Forget about intent here, how about a bug? ChatGPT often misunderstands queries, take something like ChatGPT but 100x more capable, do you really think people won't be using it to do things? And given that they will, it could easily have a bug that "oops, incinerates the atmosphere" as a side effect.
  - ivalm 4y ago
    And the monkeys thought, “how could a human be dangerous?” “Would they clobber us with stones? Surely we are stronger!” The problem is that the toolset available to someone who is much more intellectually capable is beyond what we can think of. People are not afraid of AGI because it will behave like a very smart human, people are afraid of AGI because the capability gap will be more like the one we experience between humans and other animals. The toolset available to humans when dealing with monkeys is literally incomprehensible to the monkeys. The toolset available to AGI is similarly incomprehensible to humans.
    - uh_uh 4y ago
      Part of the issue is that taking AI alignment seriously does require some level of intellectual humility — a quality that the HN comment section famously lacks.
  - mitthrowaway2 4y ago
    This has been written about in numerous places. There are multiple possible ways an AI might go about this if it saw that as its task; the probability of any one specific method being is of course lower than the total probability of the whole set. So any one method would be an unlikely and speculative scenario. The method in question could range from nuclear, chemical, biological, sabotaging agriculture, mass-producing CFCs or other pollutants, triggering wars, or other unforeseen approaches. Most scenarios allow that (a) the AGI is very smart, deceptive, creative, and resourceful, and can pose as a human or corporation to execute transactions; (b) the AGI is able to gain control over some means of funding, either legitimately or illegitimately, and thereby pay unsuspecting humans to perform seemingly-innocuous tasks like protein synthesis or package delivery; (c) you wouldn't see it coming, any more than you see the chessmate approaching several moves ahead, because the AGI would be appearing to be friendly and helpful along the way, and perhaps earning you lots of money, while it is secretly outsmarting you for its own ends. For a nuclear approach, the AI would only have to hijack the least-hackproof of US, Russian, or Chinese arsenals in order to trigger an exchange from all sides. But it would probably opt for a different method that would do less collateral damage to its own resources. This has been an issue raised since at least the early 2010s if not before, and so (arguably) predates the most recent round of US political polarization. The core arguments are unchanged, but became more urgent as AIs broke through several milestones thought to be decades out, such as defeating top human Go players, cracking the protein folding problem, and passing the Turing test with flying colors.
matusp 4y ago
I don't think they are taken that seriously. They are writing their op-eds and fiery papers, but the field marches forward nevertheless. But I agree that the whole fields seems like ivory-tower types trying to hijack the latest hottest technology an become the decision makers based on their "exprertise". AI as a field always attracted all kinds of weird individuals, probably because it is such a popular and influential concept. If we would work with their ethics, we would never allow computers to reach general population, because it is simply too dangerous and a lot of "harm" can be done. Their use will be heavily monitored and a license or an assessment would be required to use some of the functions, e.g., connect to the Internet.
- kanzure 4y ago
  We need something more like the proactionary principle instead of the precautionary principle: https://web.archive.org/web/20120203050245/http://www.maxmore.com/proactionary.html https://web.archive.org/web/20120203050245/http://www.maxmor...
- mitthrowaway2 4y ago
  I'm amazed to see how common this view is on HN. You'd think the rapid progress made in AI would give more validity and attention to these concerns raised about risk and alignment.
  - dantheman 4y ago
    The people doing this work are mediocre at best, if they could push the state of the art they'd being do that. It's a bunch of second handers trying to control those who actually understand what is going on.
    - deleted 4y ago
      [deleted]
    - mitthrowaway2 4y ago
      Can you point to any evidence or examples of this? I don't find it a convincing counterargument at all.
furyofantares 4y ago
Maybe think about the invention of nuclear weapons/energy rather than airplanes since that matches the type of concern. You can just choose not to take an airplane. If there was no manhattan project and work towards that goal had been done in the open over a long period of time, I don't think people with safety concerns would be laughed out of the building. In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes.
- flangola7 4y ago
  >In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes. This is something I've also realized. Nuclear weapons are undeniably an existential threat but they're in part mitigated because it's very very difficult to get the materials and assemble them. That was complete luck. We got lucky. That's all that saved(es) us. Imagine if making a nuke was as easy as making a steel sword. Or perhaps, as easy as running Make-Me-Global-Dictator.exe
t344344 4y ago
> but I think that if in 1900, ... serious dialogues on how to design safe air traffic control systems, you would have been laughed at. In 1900 there were massive rail stations and harbors. Just say it has to safely receive new train/ship every 10 minutes, and that very well describes the problem. > better directed at making trains or cars safer. In Greece there was train collision recently, somehow passenger and cargo train were on the same track. I doubt any advanced "lessons" were applied compared to 100 years earlier. Most people are ignorant.
kmeisthax 4y ago
The Base Assumptions undergirding all AI safety research is that: 1. AI systems respond to incentives, not commands, including incentives you did not intend to be there. 2. The technology will eventually outsmart its creators, making the imposition of reactionary controls difficult or impossible. #1 was a good bet, and has already been proven with existing ML systems. In fact, it's one of the most common training blunders you can do and there are countless examples of artificial stupidity coming about from poorly thought-out incentives. AI are ruthless optimizers that will find the loopholes in your systems and maximally exploit them[0]. #2 implies significant harm if we get it wrong (and according to #1, we will always get it wrong). It is very plausible that AI will exceed human capability in at least some tasks, since there are already problem domains where AI dominates (e.g. chess, Go, etc). My main objection to #2 is that the idea of a new structure or machine that exceeds human intelligence is not novel. Humans evolved to live in small tribes of no more than 100 people and we have not exceeded that limitation. So we're already several nested singularities deep[1] and humanity has not yet been destroyed. On the other hand, humanity itself is a singularity: the system that created us (genetic evolution) is not capable of effectively controlling us anymore. We control it, and have been doing so since the dawn of agriculture. And we can see the harms that we have done to nature in pursuit of our own goals. So it's not irrational to extrapolate and see how an AI with more intelligence than us can do more harm to us. [0] So are human children, but they do not have the I/O capability of a data center full of A100s. [1] Agriculture beget civilization, which beget nation-states, which beget corporations, which beget computers. Each one is a singularity unto it's creators.
- bootsmann 4y ago
  1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these systems are trained to do.
  - mitthrowaway2 4y ago
    One of the main reasons why AI alignment is a hard problem is that it's not easy to predict what behavior will result from optimization applied to a known loss function. (An incentive is just the gradient of the loss function, btw).
spion 4y ago
We do have "successful" (but primitive) AI systems already, at least of the kind that we consider difficult to align (value optimizers / maximizers), which have a considerable impact of the world. An example would be social media recommendation engines. We like to perscribe malice to the people spreading false news, but we forget that ultimately its often the recommendation engine that decides which things to highlight and which not to. If you give an AI a value function that measures engagement perfectly well, and you get a recommendation algorithm that spreads false news and outrage inducing posts, the algorithm did exactly what you asked from it. The problem is that the optimizer didn't optimize for other implicit values you had in mind. (This is one of the key problems discussed in AI alignment) Of course, this would be strongly contested by proponents of free speech who would say the AI gave us exactly what we wanted. But... did it? Or did the optimizer find a behavior that successfully manipulates people without taking into account the side-effect of radicalization of online discourse? Is that what we really want? So, I'd say misaligned primitive AI might be already causing serious trouble in the world.
deleted 4y ago
[deleted]
sorokod 4y ago
The counter analogy is that we don't actually know where in the timeline we are.
Sharlin 4y ago
Take AI safety and alignment concerns seriously? Industry actors are at best paying lip service to AI alignment. Currently we're firmly stuck on a timeline where permanent and drastic reduction of human potential has a much higher probability than I’m comfortable with. What do you think is an acceptable risk of human extinction in the next 50 years? 0.1%? 1%? 10%?
abeppu 4y ago
To be clear, I would claim some of the "obvious" cases where misalignment of existing organizations / institutions and society at large, and where there have been reasonable policy fixes advocated for years include: - unpenalized externalities, including carbon emissions, and other environmental effects - clearly broken tax law which allows some enormously profitable companies to pay very low tax rates - political donations protected as "speech" - partisan gerrymandering These are cases where we made systems with rules, in which agents have incentives, and where the combination of the design of the system and the incentives on participants can cause meaningful harm to the public. In many cases, transparency is meaningfully hindered, and "explanations" of particular outcomes are obscured. Several of the structural issues that AI alignment people care about are present, but the harms are already happening, and yet we seem unable (and AI alignment folk seem comparatively uninterested) in addressing them.
- mitthrowaway2 4y ago
  I don't think AI alignment folk are uninterested in addressing them. I think they do see as a red flag the fact that it's so hard even for corporations to craft incentive schemes that consistently reward good behavior without being hacked and gamed for bad results, and that's when the incentives are being followed humans who we have a lot of experience with.
dragonwriter 4y ago
> So I'm always a little surprised how seriously people take AI safety and alignment concerns about systems we don't yet know how to build Aside from Basilisk cultists and people play-acting concern for safety and alignment as a PR tactic to push centralized control of AI in support of their own financial self interests (groups which have more overlap than one might expect), the main (non-academic) safety and alignment concerns are more about the kind of systems we do know how to build, and which are seeing increasing deployment deployment demonstrable severe biases that have significant impacts in application and other issues.

by abeppu 4y ago

ditchfieldcaleb 4y ago

You're right - the odds of success are incredibly slim. But we may only have one shot, because the first mid-air collision in this analogy could literally be the end of the world.

faddypaddy34 4y ago
How exactly? Explain how an AI would cause the end of the world? Are you suggesting we would turn over all of the world's nuclear arsenals to AI to deal with? Maybe it's just me but lately it seems like everything is being labeled as "dangerous" to the point of absurdity. It seems to be following the same line as US political rhetoric where everyone is either a Nazi or communist bent on destroying the country depending on what side you generally align with.
- edanm 4y ago
  Forget the idea of an "AI" then, because the idea of "intelligence" makes the argument harder. Just think of a "new technology." Is it possible that a new technology could destroy the world? Of course. It could've turned out that nuclear weapons would incinerate the atmosphere upon detonation, as some were worried they would. It could be that the next technological innovation will kill us, there's nothing prevent it in the laws of physics. AGI is a specific technology we are worried about, because the whole premise is "once we build something that is extremely capable at a variety of things, one thing it will be capable of is destroying the world. Even by accident." We're already using AI techniques to help with problems in biology like protein folding. Take it a few dozen iterations forward, and these systems will be helping design medicines and vaccines that no human can do by themselves. At that point, what's to stop the system from creating a super-flu that kills everyone? Forget about intent here, how about a bug? ChatGPT often misunderstands queries, take something like ChatGPT but 100x more capable, do you really think people won't be using it to do things? And given that they will, it could easily have a bug that "oops, incinerates the atmosphere" as a side effect.
- ivalm 4y ago
  And the monkeys thought, “how could a human be dangerous?” “Would they clobber us with stones? Surely we are stronger!” The problem is that the toolset available to someone who is much more intellectually capable is beyond what we can think of. People are not afraid of AGI because it will behave like a very smart human, people are afraid of AGI because the capability gap will be more like the one we experience between humans and other animals. The toolset available to humans when dealing with monkeys is literally incomprehensible to the monkeys. The toolset available to AGI is similarly incomprehensible to humans.
  - uh_uh 4y ago
    Part of the issue is that taking AI alignment seriously does require some level of intellectual humility — a quality that the HN comment section famously lacks.
- mitthrowaway2 4y ago
  This has been written about in numerous places. There are multiple possible ways an AI might go about this if it saw that as its task; the probability of any one specific method being is of course lower than the total probability of the whole set. So any one method would be an unlikely and speculative scenario. The method in question could range from nuclear, chemical, biological, sabotaging agriculture, mass-producing CFCs or other pollutants, triggering wars, or other unforeseen approaches. Most scenarios allow that (a) the AGI is very smart, deceptive, creative, and resourceful, and can pose as a human or corporation to execute transactions; (b) the AGI is able to gain control over some means of funding, either legitimately or illegitimately, and thereby pay unsuspecting humans to perform seemingly-innocuous tasks like protein synthesis or package delivery; (c) you wouldn't see it coming, any more than you see the chessmate approaching several moves ahead, because the AGI would be appearing to be friendly and helpful along the way, and perhaps earning you lots of money, while it is secretly outsmarting you for its own ends. For a nuclear approach, the AI would only have to hijack the least-hackproof of US, Russian, or Chinese arsenals in order to trigger an exchange from all sides. But it would probably opt for a different method that would do less collateral damage to its own resources. This has been an issue raised since at least the early 2010s if not before, and so (arguably) predates the most recent round of US political polarization. The core arguments are unchanged, but became more urgent as AIs broke through several milestones thought to be decades out, such as defeating top human Go players, cracking the protein folding problem, and passing the Turing test with flying colors.

matusp 4y ago

I don't think they are taken that seriously. They are writing their op-eds and fiery papers, but the field marches forward nevertheless. But I agree that the whole fields seems like ivory-tower types trying to hijack the latest hottest technology an become the decision makers based on their "exprertise". AI as a field always attracted all kinds of weird individuals, probably because it is such a popular and influential concept. If we would work with their ethics, we would never allow computers to reach general population, because it is simply too dangerous and a lot of "harm" can be done. Their use will be heavily monitored and a license or an assessment would be required to use some of the functions, e.g., connect to the Internet.

kanzure 4y ago
We need something more like the proactionary principle instead of the precautionary principle: https://web.archive.org/web/20120203050245/http://www.maxmore.com/proactionary.html https://web.archive.org/web/20120203050245/http://www.maxmor...
mitthrowaway2 4y ago
I'm amazed to see how common this view is on HN. You'd think the rapid progress made in AI would give more validity and attention to these concerns raised about risk and alignment.
- dantheman 4y ago
  The people doing this work are mediocre at best, if they could push the state of the art they'd being do that. It's a bunch of second handers trying to control those who actually understand what is going on.
  - deleted 4y ago
    [deleted]
  - mitthrowaway2 4y ago
    Can you point to any evidence or examples of this? I don't find it a convincing counterargument at all.

furyofantares 4y ago

Maybe think about the invention of nuclear weapons/energy rather than airplanes since that matches the type of concern. You can just choose not to take an airplane. If there was no manhattan project and work towards that goal had been done in the open over a long period of time, I don't think people with safety concerns would be laughed out of the building. In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes.

flangola7 4y ago
>In fact I think we got kinda lucky that it turns out to be so expensive to build nukes that only nation states can do it, and that the barriers are still prohibitive even then such that we can tell if a country is trying to gain that ability. I'm not sure we could have ruled out the possibility of individuals or small groups being in control of nukes. This is something I've also realized. Nuclear weapons are undeniably an existential threat but they're in part mitigated because it's very very difficult to get the materials and assemble them. That was complete luck. We got lucky. That's all that saved(es) us. Imagine if making a nuke was as easy as making a steel sword. Or perhaps, as easy as running Make-Me-Global-Dictator.exe

t344344 4y ago

> but I think that if in 1900, ... serious dialogues on how to design safe air traffic control systems, you would have been laughed at. In 1900 there were massive rail stations and harbors. Just say it has to safely receive new train/ship every 10 minutes, and that very well describes the problem. > better directed at making trains or cars safer. In Greece there was train collision recently, somehow passenger and cargo train were on the same track. I doubt any advanced "lessons" were applied compared to 100 years earlier. Most people are ignorant.

kmeisthax 4y ago

The Base Assumptions undergirding all AI safety research is that: 1. AI systems respond to incentives, not commands, including incentives you did not intend to be there. 2. The technology will eventually outsmart its creators, making the imposition of reactionary controls difficult or impossible. #1 was a good bet, and has already been proven with existing ML systems. In fact, it's one of the most common training blunders you can do and there are countless examples of artificial stupidity coming about from poorly thought-out incentives. AI are ruthless optimizers that will find the loopholes in your systems and maximally exploit them[0]. #2 implies significant harm if we get it wrong (and according to #1, we will always get it wrong). It is very plausible that AI will exceed human capability in at least some tasks, since there are already problem domains where AI dominates (e.g. chess, Go, etc). My main objection to #2 is that the idea of a new structure or machine that exceeds human intelligence is not novel. Humans evolved to live in small tribes of no more than 100 people and we have not exceeded that limitation. So we're already several nested singularities deep[1] and humanity has not yet been destroyed. On the other hand, humanity itself is a singularity: the system that created us (genetic evolution) is not capable of effectively controlling us anymore. We control it, and have been doing so since the dawn of agriculture. And we can see the harms that we have done to nature in pursuit of our own goals. So it's not irrational to extrapolate and see how an AI with more intelligence than us can do more harm to us. [0] So are human children, but they do not have the I/O capability of a data center full of A100s. [1] Agriculture beget civilization, which beget nation-states, which beget corporations, which beget computers. Each one is a singularity unto it's creators.

bootsmann 4y ago
1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these systems are trained to do.
- mitthrowaway2 4y ago
  One of the main reasons why AI alignment is a hard problem is that it's not easy to predict what behavior will result from optimization applied to a known loss function. (An incentive is just the gradient of the loss function, btw).

spion 4y ago

We do have "successful" (but primitive) AI systems already, at least of the kind that we consider difficult to align (value optimizers / maximizers), which have a considerable impact of the world. An example would be social media recommendation engines. We like to perscribe malice to the people spreading false news, but we forget that ultimately its often the recommendation engine that decides which things to highlight and which not to. If you give an AI a value function that measures engagement perfectly well, and you get a recommendation algorithm that spreads false news and outrage inducing posts, the algorithm did exactly what you asked from it. The problem is that the optimizer didn't optimize for other implicit values you had in mind. (This is one of the key problems discussed in AI alignment) Of course, this would be strongly contested by proponents of free speech who would say the AI gave us exactly what we wanted. But... did it? Or did the optimizer find a behavior that successfully manipulates people without taking into account the side-effect of radicalization of online discourse? Is that what we really want? So, I'd say misaligned primitive AI might be already causing serious trouble in the world.

deleted 4y ago

[deleted]

sorokod 4y ago

The counter analogy is that we don't actually know where in the timeline we are.

Sharlin 4y ago

Take AI safety and alignment concerns seriously? Industry actors are at best paying lip service to AI alignment. Currently we're firmly stuck on a timeline where permanent and drastic reduction of human potential has a much higher probability than I’m comfortable with. What do you think is an acceptable risk of human extinction in the next 50 years? 0.1%? 1%? 10%?

abeppu 4y ago

To be clear, I would claim some of the "obvious" cases where misalignment of existing organizations / institutions and society at large, and where there have been reasonable policy fixes advocated for years include: - unpenalized externalities, including carbon emissions, and other environmental effects - clearly broken tax law which allows some enormously profitable companies to pay very low tax rates - political donations protected as "speech" - partisan gerrymandering These are cases where we made systems with rules, in which agents have incentives, and where the combination of the design of the system and the incentives on participants can cause meaningful harm to the public. In many cases, transparency is meaningfully hindered, and "explanations" of particular outcomes are obscured. Several of the structural issues that AI alignment people care about are present, but the harms are already happening, and yet we seem unable (and AI alignment folk seem comparatively uninterested) in addressing them.

mitthrowaway2 4y ago
I don't think AI alignment folk are uninterested in addressing them. I think they do see as a red flag the fact that it's so hard even for corporations to craft incentive schemes that consistently reward good behavior without being hacked and gamed for bad results, and that's when the incentives are being followed humans who we have a lot of experience with.

dragonwriter 4y ago

> So I'm always a little surprised how seriously people take AI safety and alignment concerns about systems we don't yet know how to build Aside from Basilisk cultists and people play-acting concern for safety and alignment as a PR tactic to push centralized control of AI in support of their own financial self interests (groups which have more overlap than one might expect), the main (non-academic) safety and alignment concerns are more about the kind of systems we do know how to build, and which are seeing increasing deployment deployment demonstrable severe biases that have significant impacts in application and other issues.