12 ms·

> Like yes, we are able to think of thousands of hypothetical ways technology (even those inferior to full AGI) could go off the rails in a catastrophic way and

by gretch 2y ago

> Like yes, we are able to think of thousands of hypothetical ways technology (even those inferior to full AGI) could go off the rails in a catastrophic way and post and discuss these scenarios endlessly... and yet it doesn't result in a slowing or stopping of the progress leading there. The problem is sifting through all of the doomsayer false positives to get to any amount of cogent advice. At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. And if we had taken their "lesson", then human society would be in a much worse place. Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect.

beezlebroxxxxxx 2y ago
> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. All of the focus on AGI is a distraction. I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas, which is, in my opinion, an incredibly naïve notion. I would rather have a state say "we won't use this technology for evil" than a state that says nothing at all and simply allows the businesses to develop in any direction their greed leads them. It's entirely valid to critique the uses of a technology, because "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) is a technology like any other, like a landmine, like a synthetic virus, etc. In the same way, it's valid to criticize an actor for purposely hiding their intentions with a technology.
- circuit10 2y ago
  The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. In fact we humans are doing the same thing, we can't really improve our intelligence directly but we are trying to create AI to achieve our goals, and there's no reason that the AI itself wouldn't do so assuming it's capable and we don't attempt to stop it, and currently we don't really know how to reliably control it. We have absolutely no idea how to specify human values in a robust way which is what we would need to figure out to build this safely
  - slg 2y ago
    I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goals that don't align with "human values". We don't need to invent hypothetical sentient AI boogeymen turning the universe into paperclips in order to be fearful of the future that ubiquitous AI creates. Humans would happily do that too if they get to preside over that paperclip empire.
    - Filligree 2y ago
      “Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
      - slg 2y ago
        I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
        circuit10 2y ago
        "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
        wombatpm 2y ago
        Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
- roenxi 2y ago
  But if the state approaches a technology with intent it is usually for the purposes of a military offence. I don't think that is a good idea in the context of AI! Although I also don't think there is any stopping it. The US has things like DARPA for example and a lot of Chinese investment seems to be done with the intent of providing capabilities to their army. The list of things states have attempted to deploy offensively is nearly endless. Modern operations research arguably came out of the British empire attempting (succeeding) to weaponise mathematics. If you give a state fertiliser it makes bombs, if you give it nuclear power it makes bombs, if you give it drones it makes bombs, if you give it advanced science or engineering of any form it makes bombs. States are the most ingenious system for turning things into bombs that we've ever invented; in the grand old days of siege warfare they even managed to weaponise corpses, refuse and junk because it turned out lobbing that stuff at the enemy was effective. The entire spectrum of technology from nothing to nanotech, hurled at enemies to kill them. We'd all love if states commit to not doing evil but the state is the entity most active at figuring out how to use new tech X for evil.
  - Agentus 2y ago
    quick devil’s advocate on a tangential point. is designer better killing tools necessarily evil? seems like the nature of the world is eat or be eaten and on the empire-scale, conquer or be conquered. that latter point seems to be the historical norm. Even with democracy, reasoning doesnt prevail but force of numbers seems to be the end determiner. Point is, humans arent easy to reason with or negotiate, coercion is the dominant force through out history especially when dealing with groups of different values. if one groups gives up the arms race of ultimate coercion tools or loses a conflict then they become subservient to the winners terms and norms (japan, germany, even Britain and France plus all the smaller states in between are subservient to the US)
    - musicale 2y ago
      > is design[ing] better killing tools necessarily evil? Who could possibly have predicted that the autonomous, invincible doomsday weapon we created for the good of humanity might one day be used against us?
- bbor 2y ago
  I usually don't engage on A[GS]I on here, but I feel like this is a decent time for an exception -- you're certainly well spoken and clear, which helps! Three things: (I) All of the focus on AGI is a distraction. I strongly disagree on that, at least if you're implying some intentionality. I think it's just provably true that many experts are honestly worried, even if you don't include the people who have dedicated a good portion of their lives to the cause. For example: OpenAI has certainly been corrupted through the loss of its nonprofit board, but I think their founding charter[1] was pretty clearly earnest -- and dire. (II) "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) To be fair, this uncertainty in the term has been there since the dawn of the field, a fact made clear by perrenial rephrasings of the sentiment "AI is whatever hasn't been done yet" (~Larry Tesler 1979, see [2]). I'd love to get into the weeds on the different kinds of intelligence and why being to absolutist about the term can get real Faustian real quick, but these quotes bring up a more convincing, fundamental point: these chatbots are damn impressive. They do something--intuitive inference+fluent language use--that was impossible yesterday, and many experts would've guessed was decades away at least, if not centuries. Truly intelligent or not on their own, that's a more important development than you imply here. Finally, that brings me to the crux: (III) AI... is a technology like any other There's a famous Sundar Pichai (Google CEO) quote that he's been paraphrasing since 2018 -- soon after ChatGPT broke, he phrased it as such: I’ve always thought of A.I. as the most profound technology humanity is working on-more profound than fire or electricity or anything that we’ve done in the past. It gets to the essence of what intelligence is, what humanity is. We are developing technology which, for sure, one day will be far more capable than anything we’ve ever seen before. [3] When skeptics hear this, they understandably tend to write this off as capitalist bias from someone trying to pump Google's stock. However, I'd retort: 1) This kind of talk is so grandiose that it seems like a questionable move if that's the goal, 2) it's a sentiment echoed by many scientists (as I mentioned at the start of this rant) and 3) the unprecedented investments made across the world into the DL boom speak for themselves, sincerity-wise. Yes, this is because AI will create uber-efficient factories, upset labor relations, produce terrifying autonomous weapons, and all that stuff we're used to hearing about from the likes of Bostrom[4], Yudkowsky[5], and my personal fave, Huw Price[6]. But Pichai's raising something even more fundamental: the prospect of artificial people. Even if we ignore the I-Robot-style concerns about their potential moral standing, that is just a fundamentally spooky prospect, bringing very fundamental questions of A) individual worth and B) the nature of human cognition to the fore. And, to circle back: distinct from anything we've seen before. To close this long anxiety-driven manuscript, I'll end with a quote from an underappreciated philosopher of technology named Lewis Mumford on what he called "neotechnics": The scientific method, whose chief advances had been in mathematics and the physical sciences, took possession of other domains of experience: the living organism and human society also became the objects of systematic investigation... instead of mechanism forming a pattern for life, living organisms began to form a pattern for mechanism. In short, the concepts of science--hitherto associated largely with the cosmic, the inorganic, the "mechanical"--were now applied to every phase of human experience and every manifestation of life... men sought for an underlying order and logic of events which would embrace more complex manifestations.[7] TL;DR: IMHO, the US & UK refusing to cooperate at this critical moment is the most important event of your lifetime so far. [1] OpenAI's Charter https://web.archive.org/web/20230714043611/https://openai.com/charter https://web.archive.org/web/20230714043611/https://openai.co... [2] Investigation of a famous AI quote https://quoteinvestigator.com/2024/06/20/not-ai/ https://quoteinvestigator.com/2024/06/20/not-ai/ [3] Pichai, 2023: "AI is more profound than fire or electricity" https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profound-than-fire-electricity/ https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profou... [4] Bostrom, 2014: Superintelligence https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... [5] Yudkowsky, 2013: Intelligence Explosion Microeconomics https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [6] Huw Price's bio @ The Center for Existential Risk https://www.cser.ac.uk/team/huw-price/ https://www.cser.ac.uk/team/huw-price/ [7] Mumford, 1934: Technics and Civilization https://archive.org/details/in.ernet.dli.2015.49974 https://archive.org/details/in.ernet.dli.2015.49974
- jstanley 2y ago
  > I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas The sleight of hand here is the implication that human interactions, values, and ideas are only expressed through the state.
  - forgetfreeman 2y ago
    The sleight of hand here is implying that there are any forces smaller than nation states that can credibly reign in problematic technology. Relying on good intentions to win out against market forces isn't even naive, it's just stupid.
    - TeMPOraL 2y ago
      So many sleights here. Another sleight of hand in this subthread is suggesting that "the idea that technology advances autonomously, independent of human interactions, values, or ideas" is merely an idea, and not an actual observable fact at scale. Society and culture are downstream of economics, and economics is mostly downstream of technological progress. Of course, the progress isn't autonomous in the sense of having a sentient mind of its own - it's "merely" gradient descent down the economic landscape. Just like the market itself. There's no reining in of problematic technology unless, like you say, nation states get involved directly. And they don't stand much chance either unless they get serious. People still laugh at Eliezer's comments from that news article of yesteryear, but he was and is spot-on: being serious about restricting technology actually does mean threatening to drop bombs on facilities developing it in violation of restrictions - if we're not ready to have our representatives make such threats, and then actually follow through and drop the bombs if someone decides to test our resolve, then we're not serious.
      - forgetfreeman 2y ago
        People laugh at all kinds of common sense declarations that they ought not find funny in the slightest. Its one of our species glaring failures.
RajT88 2y ago
A useful counterexample is all the people who predicted doomsday scenarios with the advent of nuclear weapons. Just because it has not come to pass yet does not mean they were wrong. We have come close to nuclear annihilation several times. We may yet, with or without AI.
- gretch 2y ago
  >Just because it has not come to pass yet does not mean they were wrong. This assertion is meaningless because it can be applied to anything. "I think vaccines cause autism and will cause human annihilation" - just because it has not yet come to pass does not mean it is wrong.
  - anigbrowl 2y ago
    No. there have not been any nuclear exchanges, whereas there have been millions, probably billions of vaccinations. You're giving equal weight to conjecture and empirical data.
    - RajT88 2y ago
      There have been tens of billions of vaccinations.
- idontwantthis 2y ago
  And imagine if private companies had had the resources to develop nuclear weapons and the US government had decided it didn’t need to even regulate them.
  - greentxt 2y ago
    A future that may yet come.
    - RajT88 2y ago
      The Onion just had a funny video where Lakewood Church conducted a nuclear test, firing a missile over Washington. True to form, it was deadpan, and featured Joel Olsteen as a Kim Jung Un type leader.
gibspaulding 2y ago
> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. And if we had taken their "lesson", then human society would be in a much worse place. In the long run the invention of the printing press was undoubtedly a good thing, but it is worth noting that in the century following the spread of the printing press basically every country in Europe had some sort of revolution. It seems likely that “Interesting Times” may lay ahead.
- llm_trw 2y ago
  They had some sort of revolution the previous few centuries too. Pretending that Europe wasn't in a perpetual blood bath since the end of the Pax Romana until 1815 shows a gross ignorance of basic facts. The printing press was a net positive in every time scale.
  - achierius 2y ago
    I'm sorry but that's just false. > Pretending that Europe wasn't in a perpetual blood bath since the end of the Pax Romana until 1815 shows a gross ignorance of basic facts. This shows that your understanding of history is rooted in pop-culture, not reality. What "revolutions" were there in France between the ascension of Hugh Capet and the European Wars of Religion? Through that whole period the Capetian Dynasty stayed in power. Or in Scandinavia -- from Christianization on the three kingdoms were shockingly stable. Even in the Holy Roman Empire -- none of the petty revolts, rebellions, or succession disputes came close to the magnitude of carnage wrought by the 30 Year's War. This we know both from demographic studies and the reports of contemporaries.
- daedrdev 2y ago
  Given countries at the time were all monarchies with limited rights, I'm not sure if it's too comparable.
harrall 2y ago
But we already know. I think people arguing about AI being good versus bad are wasting their breath. Both sides are equally right. History tells us the industrial revolution both revolutionized humanity’s relative quality of life while also ruining a lot of people’s livelihood in one fell swoop. We also know there was nothing we could do to stop it. What advice can we can take from it? I don’t know. Life both rocks and sucks at the same time. You kind of just take things day by day and do your best to adapt for both yourself and everyone around you.
- radley 2y ago
  > What advice can we can take from it? That we often won't have control over big changes affecting our lives, so be prepared. If possible, get out in front and ride the wave. If not, duck under and don't let it churn you up too much.
  - harrall 2y ago
    That would be the adaptation I’m talking about.
  - Angostura 2y ago
    This one is a tsunami though. I have absolutely no idea how to either ride it or duck under it. It's my kids that I'm worried about largely - currently finishing up their degrees at university
    - onemoresoop 2y ago
      It's exactly what I'm worried most about too, the kids. I have younger ones. We had a good ride thus far but they don't seem so lucky, things look pretty badly overall without an obvious for much improvement any time soon.
    - radley 2y ago
      I don't entirely agree. Internet was a tsunami. Mobile was a tsunami. Both seemed impactful at first, but we didn't know exactly how right away. We all figured it out and adopted, some better than others. Schools are way ahead of us. Your kids are already using AI in their academic environments. I'd only be worried if they're not.
ls612 2y ago
>At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. And if we had taken their "lesson", then human society would be in a much worse place. One could argue that the printing press did radically upset the existing geopolitical order of the late 15th century and led to early modern Europe suffering the worst spate of warfare and devastation it would see until the 20th century. The doomsayers back then predicting centuries of death and war and turmoil were right, yet from our position 550 years later we obviously think the printing press is a good thing. I wonder what people in 2300 will say about networked computers...
pdpi 2y ago
> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. One thing that should be completely obvious by now is that the current wave of generative AI is highly asymmetric. It's shockingly more powerful in the hands of grifters (who are happy to monetise vast amounts of slop) or state-level bad actors (whose propaganda isn't impeded by hallucinations generating lies) than it is in the hands of the "good guys" who are hampered by silly things like principles.
golergka 2y ago
> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. Printing press put Europe into a couple centuries of bloody religious wars. They were not wrong.
dartos 2y ago
> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. What energy? What were they wrong about? The luddite type groups have historically been correct in their fears. It just didn’t matter in the face of industrialization.
BurningFrog 2y ago
The printing press meant regular people could read the bible, which led to protestantism and a century of very bloody wars across Europe. Since the victors write history we now think the end result was great. but for a lot of people the world they loved was torn to bloody pieces. Something similar can happen with AI. In the end, whoever wins the wars will declare that the new world is awesome. But it might not be what you or me (may we rest in peace) would agree with.
deleted 2y ago
[deleted]
TeMPOraL 2y ago
> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. Were they? The first thing the printing press did was to break Christianity. It's what made attempts at reforming the Catholic Church finally stick, enabling what we now call Reformation to happen. Reformation forever broke Christianity into pieces, and in the process it started a bunch of religious wars in Europe, as well as tons of neighborly carnage. > And if we had taken their "lesson", then human society would be in a much worse place. Was the invention of the printing press a net good for humanity? Most certainly so, looking back from today. Did people living back then knew what they were getting into? Not really. And since their share of the fruits of that invention was mostly bloodshed, job loss, and shattering of the world order they knew, I wouldn't blame them from being pissed off about getting the short end of the stick, and perhaps looking for ways to undo it. I'm starting to think that talking about inventions as good or bad (or the cop-out, "dual use") is bad framing. Rather, it seems to me that every major invention will eventually turn out beneficial[0], but introducing an invention always first extracts a cost in blood. Be it fire or printing press or atomic bomb, a lot of people end up suffering and dying before societies eventually figure out how to handle the new thing and do some good with it. I'm very much in favor of progress, but I understand the fear. No matter the ultimate benefits, we are the generation that cough up blood as payment for AI/AGI, and it ain't gonna be pleasant. -- [0] - Assuming they don't kill us first - see AGI.
- shmeeed 2y ago
  That's a very thought provoking insight regarding to the often repeated "printing press doomsayer" talking point. Thank you!
- throwawayqqq11 2y ago
  I dont understand, why any highly sophisticated AI should invest that much resources to kill us instead of investing it to relocating and protecting itself. Yes, ants could technically conspire to sneak up to you while you sleep and bite you all at once to kill you, so do you go out to eradicate all ants?
  - 2y ago
concordDance 2y ago
Why are you comparing AGI (which we do not have yet and do not know hoe to get) to the printing press rather than comparing it to the evolution of humans? Actual proper as-smart-as-a-human-except-where-its-smarter copy-pasteable intelligence is not a tool, its a new species. One that can replicate and evolve orders of magnitude faster. I've no idea when this will appear, but once it does, the extinction risk is extreme. Best case scenario is us going the way of the chimpanzee, kept in little nature reserves and occasionally as pets. Worst case scenario is going the way of the mammoth.
sandworm101 2y ago
>> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. Valid or not, it does not matter. AI development is not in the hands of everyday people. We have zero input into how it will be used. Our opinions re its dangers are irrelevant to those who believe it the next golden goose. They will push it as far as physically possible to wring every penny of profitability. Everything else is of trivial consequence.
tgv 2y ago
> The problem is sifting through all of the doomsayer false positives to get to any amount of cogent advice. Why? Because we don't understand the risk. And apparently, that's enough reason to go ahead for the regulation-averse tech mind set. But it isn't. We've had enough problems in the past to understand that, and it's not as if pushing ahead is critical in this case. Would this address climate change, the balance between risk and reward could be different, but "AI" simply doesn't have that urgency. It only has urgency for those that want to become rich out of being first.
lores 2y ago
It's not just AI/AGI, it's its mixing with the current climate of unlimited greed, disappearance of even the pretense of a social contract, and the vast surveillance powers available. Technological dictatorship, that's what's most worrying. I love dystopian cyberpunk, but I want it to stay in books.

by gretch 2y ago

beezlebroxxxxxx 2y ago

> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. All of the focus on AGI is a distraction. I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas, which is, in my opinion, an incredibly naïve notion. I would rather have a state say "we won't use this technology for evil" than a state that says nothing at all and simply allows the businesses to develop in any direction their greed leads them. It's entirely valid to critique the uses of a technology, because "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) is a technology like any other, like a landmine, like a synthetic virus, etc. In the same way, it's valid to criticize an actor for purposely hiding their intentions with a technology.

circuit10 2y ago
The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. In fact we humans are doing the same thing, we can't really improve our intelligence directly but we are trying to create AI to achieve our goals, and there's no reason that the AI itself wouldn't do so assuming it's capable and we don't attempt to stop it, and currently we don't really know how to reliably control it. We have absolutely no idea how to specify human values in a robust way which is what we would need to figure out to build this safely
- slg 2y ago
  I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goals that don't align with "human values". We don't need to invent hypothetical sentient AI boogeymen turning the universe into paperclips in order to be fearful of the future that ubiquitous AI creates. Humans would happily do that too if they get to preside over that paperclip empire.
  - Filligree 2y ago
    “Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
    - slg 2y ago
      I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
      - circuit10 2y ago
        "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
        wombatpm 2y ago
        Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
roenxi 2y ago
But if the state approaches a technology with intent it is usually for the purposes of a military offence. I don't think that is a good idea in the context of AI! Although I also don't think there is any stopping it. The US has things like DARPA for example and a lot of Chinese investment seems to be done with the intent of providing capabilities to their army. The list of things states have attempted to deploy offensively is nearly endless. Modern operations research arguably came out of the British empire attempting (succeeding) to weaponise mathematics. If you give a state fertiliser it makes bombs, if you give it nuclear power it makes bombs, if you give it drones it makes bombs, if you give it advanced science or engineering of any form it makes bombs. States are the most ingenious system for turning things into bombs that we've ever invented; in the grand old days of siege warfare they even managed to weaponise corpses, refuse and junk because it turned out lobbing that stuff at the enemy was effective. The entire spectrum of technology from nothing to nanotech, hurled at enemies to kill them. We'd all love if states commit to not doing evil but the state is the entity most active at figuring out how to use new tech X for evil.
- Agentus 2y ago
  quick devil’s advocate on a tangential point. is designer better killing tools necessarily evil? seems like the nature of the world is eat or be eaten and on the empire-scale, conquer or be conquered. that latter point seems to be the historical norm. Even with democracy, reasoning doesnt prevail but force of numbers seems to be the end determiner. Point is, humans arent easy to reason with or negotiate, coercion is the dominant force through out history especially when dealing with groups of different values. if one groups gives up the arms race of ultimate coercion tools or loses a conflict then they become subservient to the winners terms and norms (japan, germany, even Britain and France plus all the smaller states in between are subservient to the US)
  - musicale 2y ago
    > is design[ing] better killing tools necessarily evil? Who could possibly have predicted that the autonomous, invincible doomsday weapon we created for the good of humanity might one day be used against us?
bbor 2y ago
I usually don't engage on A[GS]I on here, but I feel like this is a decent time for an exception -- you're certainly well spoken and clear, which helps! Three things: (I) All of the focus on AGI is a distraction. I strongly disagree on that, at least if you're implying some intentionality. I think it's just provably true that many experts are honestly worried, even if you don't include the people who have dedicated a good portion of their lives to the cause. For example: OpenAI has certainly been corrupted through the loss of its nonprofit board, but I think their founding charter[1] was pretty clearly earnest -- and dire. (II) "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) To be fair, this uncertainty in the term has been there since the dawn of the field, a fact made clear by perrenial rephrasings of the sentiment "AI is whatever hasn't been done yet" (~Larry Tesler 1979, see [2]). I'd love to get into the weeds on the different kinds of intelligence and why being to absolutist about the term can get real Faustian real quick, but these quotes bring up a more convincing, fundamental point: these chatbots are damn impressive. They do something--intuitive inference+fluent language use--that was impossible yesterday, and many experts would've guessed was decades away at least, if not centuries. Truly intelligent or not on their own, that's a more important development than you imply here. Finally, that brings me to the crux: (III) AI... is a technology like any other There's a famous Sundar Pichai (Google CEO) quote that he's been paraphrasing since 2018 -- soon after ChatGPT broke, he phrased it as such: I’ve always thought of A.I. as the most profound technology humanity is working on-more profound than fire or electricity or anything that we’ve done in the past. It gets to the essence of what intelligence is, what humanity is. We are developing technology which, for sure, one day will be far more capable than anything we’ve ever seen before. [3] When skeptics hear this, they understandably tend to write this off as capitalist bias from someone trying to pump Google's stock. However, I'd retort: 1) This kind of talk is so grandiose that it seems like a questionable move if that's the goal, 2) it's a sentiment echoed by many scientists (as I mentioned at the start of this rant) and 3) the unprecedented investments made across the world into the DL boom speak for themselves, sincerity-wise. Yes, this is because AI will create uber-efficient factories, upset labor relations, produce terrifying autonomous weapons, and all that stuff we're used to hearing about from the likes of Bostrom[4], Yudkowsky[5], and my personal fave, Huw Price[6]. But Pichai's raising something even more fundamental: the prospect of artificial people. Even if we ignore the I-Robot-style concerns about their potential moral standing, that is just a fundamentally spooky prospect, bringing very fundamental questions of A) individual worth and B) the nature of human cognition to the fore. And, to circle back: distinct from anything we've seen before. To close this long anxiety-driven manuscript, I'll end with a quote from an underappreciated philosopher of technology named Lewis Mumford on what he called "neotechnics": The scientific method, whose chief advances had been in mathematics and the physical sciences, took possession of other domains of experience: the living organism and human society also became the objects of systematic investigation... instead of mechanism forming a pattern for life, living organisms began to form a pattern for mechanism. In short, the concepts of science--hitherto associated largely with the cosmic, the inorganic, the "mechanical"--were now applied to every phase of human experience and every manifestation of life... men sought for an underlying order and logic of events which would embrace more complex manifestations.[7] TL;DR: IMHO, the US & UK refusing to cooperate at this critical moment is the most important event of your lifetime so far. [1] OpenAI's Charter https://web.archive.org/web/20230714043611/https://openai.com/charter https://web.archive.org/web/20230714043611/https://openai.co... [2] Investigation of a famous AI quote https://quoteinvestigator.com/2024/06/20/not-ai/ https://quoteinvestigator.com/2024/06/20/not-ai/ [3] Pichai, 2023: "AI is more profound than fire or electricity" https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profound-than-fire-electricity/ https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profou... [4] Bostrom, 2014: Superintelligence https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... [5] Yudkowsky, 2013: Intelligence Explosion Microeconomics https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [6] Huw Price's bio @ The Center for Existential Risk https://www.cser.ac.uk/team/huw-price/ https://www.cser.ac.uk/team/huw-price/ [7] Mumford, 1934: Technics and Civilization https://archive.org/details/in.ernet.dli.2015.49974 https://archive.org/details/in.ernet.dli.2015.49974
jstanley 2y ago
> I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas The sleight of hand here is the implication that human interactions, values, and ideas are only expressed through the state.
- forgetfreeman 2y ago
  The sleight of hand here is implying that there are any forces smaller than nation states that can credibly reign in problematic technology. Relying on good intentions to win out against market forces isn't even naive, it's just stupid.
  - TeMPOraL 2y ago
    So many sleights here. Another sleight of hand in this subthread is suggesting that "the idea that technology advances autonomously, independent of human interactions, values, or ideas" is merely an idea, and not an actual observable fact at scale. Society and culture are downstream of economics, and economics is mostly downstream of technological progress. Of course, the progress isn't autonomous in the sense of having a sentient mind of its own - it's "merely" gradient descent down the economic landscape. Just like the market itself. There's no reining in of problematic technology unless, like you say, nation states get involved directly. And they don't stand much chance either unless they get serious. People still laugh at Eliezer's comments from that news article of yesteryear, but he was and is spot-on: being serious about restricting technology actually does mean threatening to drop bombs on facilities developing it in violation of restrictions - if we're not ready to have our representatives make such threats, and then actually follow through and drop the bombs if someone decides to test our resolve, then we're not serious.
    - forgetfreeman 2y ago
      People laugh at all kinds of common sense declarations that they ought not find funny in the slightest. Its one of our species glaring failures.

RajT88 2y ago

A useful counterexample is all the people who predicted doomsday scenarios with the advent of nuclear weapons. Just because it has not come to pass yet does not mean they were wrong. We have come close to nuclear annihilation several times. We may yet, with or without AI.

gretch 2y ago
>Just because it has not come to pass yet does not mean they were wrong. This assertion is meaningless because it can be applied to anything. "I think vaccines cause autism and will cause human annihilation" - just because it has not yet come to pass does not mean it is wrong.
- anigbrowl 2y ago
  No. there have not been any nuclear exchanges, whereas there have been millions, probably billions of vaccinations. You're giving equal weight to conjecture and empirical data.
  - RajT88 2y ago
    There have been tens of billions of vaccinations.
idontwantthis 2y ago
And imagine if private companies had had the resources to develop nuclear weapons and the US government had decided it didn’t need to even regulate them.
- greentxt 2y ago
  A future that may yet come.
  - RajT88 2y ago
    The Onion just had a funny video where Lakewood Church conducted a nuclear test, firing a missile over Washington. True to form, it was deadpan, and featured Joel Olsteen as a Kim Jung Un type leader.

gibspaulding 2y ago

> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. And if we had taken their "lesson", then human society would be in a much worse place. In the long run the invention of the printing press was undoubtedly a good thing, but it is worth noting that in the century following the spread of the printing press basically every country in Europe had some sort of revolution. It seems likely that “Interesting Times” may lay ahead.

llm_trw 2y ago
They had some sort of revolution the previous few centuries too. Pretending that Europe wasn't in a perpetual blood bath since the end of the Pax Romana until 1815 shows a gross ignorance of basic facts. The printing press was a net positive in every time scale.
- achierius 2y ago
  I'm sorry but that's just false. > Pretending that Europe wasn't in a perpetual blood bath since the end of the Pax Romana until 1815 shows a gross ignorance of basic facts. This shows that your understanding of history is rooted in pop-culture, not reality. What "revolutions" were there in France between the ascension of Hugh Capet and the European Wars of Religion? Through that whole period the Capetian Dynasty stayed in power. Or in Scandinavia -- from Christianization on the three kingdoms were shockingly stable. Even in the Holy Roman Empire -- none of the petty revolts, rebellions, or succession disputes came close to the magnitude of carnage wrought by the 30 Year's War. This we know both from demographic studies and the reports of contemporaries.
daedrdev 2y ago
Given countries at the time were all monarchies with limited rights, I'm not sure if it's too comparable.

harrall 2y ago

But we already know. I think people arguing about AI being good versus bad are wasting their breath. Both sides are equally right. History tells us the industrial revolution both revolutionized humanity’s relative quality of life while also ruining a lot of people’s livelihood in one fell swoop. We also know there was nothing we could do to stop it. What advice can we can take from it? I don’t know. Life both rocks and sucks at the same time. You kind of just take things day by day and do your best to adapt for both yourself and everyone around you.

radley 2y ago
> What advice can we can take from it? That we often won't have control over big changes affecting our lives, so be prepared. If possible, get out in front and ride the wave. If not, duck under and don't let it churn you up too much.
- harrall 2y ago
  That would be the adaptation I’m talking about.
- Angostura 2y ago
  This one is a tsunami though. I have absolutely no idea how to either ride it or duck under it. It's my kids that I'm worried about largely - currently finishing up their degrees at university
  - onemoresoop 2y ago
    It's exactly what I'm worried most about too, the kids. I have younger ones. We had a good ride thus far but they don't seem so lucky, things look pretty badly overall without an obvious for much improvement any time soon.
  - radley 2y ago
    I don't entirely agree. Internet was a tsunami. Mobile was a tsunami. Both seemed impactful at first, but we didn't know exactly how right away. We all figured it out and adopted, some better than others. Schools are way ahead of us. Your kids are already using AI in their academic environments. I'd only be worried if they're not.

ls612 2y ago

>At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. And if we had taken their "lesson", then human society would be in a much worse place. One could argue that the printing press did radically upset the existing geopolitical order of the late 15th century and led to early modern Europe suffering the worst spate of warfare and devastation it would see until the 20th century. The doomsayers back then predicting centuries of death and war and turmoil were right, yet from our position 550 years later we obviously think the printing press is a good thing. I wonder what people in 2300 will say about networked computers...

pdpi 2y ago

> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. One thing that should be completely obvious by now is that the current wave of generative AI is highly asymmetric. It's shockingly more powerful in the hands of grifters (who are happy to monetise vast amounts of slop) or state-level bad actors (whose propaganda isn't impeded by hallucinations generating lies) than it is in the hands of the "good guys" who are hampered by silly things like principles.

golergka 2y ago

> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. Printing press put Europe into a couple centuries of bloody religious wars. They were not wrong.

dartos 2y ago

> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. What energy? What were they wrong about? The luddite type groups have historically been correct in their fears. It just didn’t matter in the face of industrialization.

BurningFrog 2y ago

The printing press meant regular people could read the bible, which led to protestantism and a century of very bloody wars across Europe. Since the victors write history we now think the end result was great. but for a lot of people the world they loved was torn to bloody pieces. Something similar can happen with AI. In the end, whoever wins the wars will declare that the new world is awesome. But it might not be what you or me (may we rest in peace) would agree with.

deleted 2y ago

[deleted]

TeMPOraL 2y ago

> At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. Were they? The first thing the printing press did was to break Christianity. It's what made attempts at reforming the Catholic Church finally stick, enabling what we now call Reformation to happen. Reformation forever broke Christianity into pieces, and in the process it started a bunch of religious wars in Europe, as well as tons of neighborly carnage. > And if we had taken their "lesson", then human society would be in a much worse place. Was the invention of the printing press a net good for humanity? Most certainly so, looking back from today. Did people living back then knew what they were getting into? Not really. And since their share of the fruits of that invention was mostly bloodshed, job loss, and shattering of the world order they knew, I wouldn't blame them from being pissed off about getting the short end of the stick, and perhaps looking for ways to undo it. I'm starting to think that talking about inventions as good or bad (or the cop-out, "dual use") is bad framing. Rather, it seems to me that every major invention will eventually turn out beneficial[0], but introducing an invention always first extracts a cost in blood. Be it fire or printing press or atomic bomb, a lot of people end up suffering and dying before societies eventually figure out how to handle the new thing and do some good with it. I'm very much in favor of progress, but I understand the fear. No matter the ultimate benefits, we are the generation that cough up blood as payment for AI/AGI, and it ain't gonna be pleasant. -- [0] - Assuming they don't kill us first - see AGI.

shmeeed 2y ago
That's a very thought provoking insight regarding to the often repeated "printing press doomsayer" talking point. Thank you!
throwawayqqq11 2y ago
I dont understand, why any highly sophisticated AI should invest that much resources to kill us instead of investing it to relocating and protecting itself. Yes, ants could technically conspire to sneak up to you while you sleep and bite you all at once to kill you, so do you go out to eradicate all ants?
- 2y ago

concordDance 2y ago

Why are you comparing AGI (which we do not have yet and do not know hoe to get) to the printing press rather than comparing it to the evolution of humans? Actual proper as-smart-as-a-human-except-where-its-smarter copy-pasteable intelligence is not a tool, its a new species. One that can replicate and evolve orders of magnitude faster. I've no idea when this will appear, but once it does, the extinction risk is extreme. Best case scenario is us going the way of the chimpanzee, kept in little nature reserves and occasionally as pets. Worst case scenario is going the way of the mammoth.

sandworm101 2y ago

>> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. Valid or not, it does not matter. AI development is not in the hands of everyday people. We have zero input into how it will be used. Our opinions re its dangers are irrelevant to those who believe it the next golden goose. They will push it as far as physically possible to wring every penny of profitability. Everything else is of trivial consequence.

tgv 2y ago

> The problem is sifting through all of the doomsayer false positives to get to any amount of cogent advice. Why? Because we don't understand the risk. And apparently, that's enough reason to go ahead for the regulation-averse tech mind set. But it isn't. We've had enough problems in the past to understand that, and it's not as if pushing ahead is critical in this case. Would this address climate change, the balance between risk and reward could be different, but "AI" simply doesn't have that urgency. It only has urgency for those that want to become rich out of being first.

lores 2y ago

It's not just AI/AGI, it's its mixing with the current climate of unlimited greed, disappearance of even the pretense of a social contract, and the vast surveillance powers available. Technological dictatorship, that's what's most worrying. I love dystopian cyberpunk, but I want it to stay in books.