4 ms·

Not Without Us – Joseph Weizenbaum (1986)

64 points by drcwpl 3y ago

nonrandomstring 3y ago
This has been floating around HN for two days and hasn't received any comments, but it's an insanely prescient paper much apropos the 1800+ comments in threads on "video AI".
ebcode 3y ago
Thanks for posting this. For those unaware, Weizenbaum wrote one of the first chatbots, ELIZA. https://en.wikipedia.org/wiki/Joseph_Weizenbaum https://en.wikipedia.org/wiki/Joseph_Weizenbaum
dyoo1979 3y ago
Thank you for bringing this up. I had not read this before, but it speaks to my deep concerns.
neilv 3y ago
> [...] the building next to the one in which I work, the world famous Draper Laboratory. [...] devoted almost entirely to missile guidance and submarine navigation. In front of that Draper building, there was a strange pillar, which I liked to imagine housed a geo-positioning reference point -- perhaps even the origin of a coordinate system. Then, if ever there were an ICBM guidance failure, on that fateful day, Draper might be ground (0,0,0). Like an engineer standing under the bridge they built, as the first trucks drive over it. I was thinking of that as intentional/conscious by the engineers. (BTW, I'm not criticizing; only a little dark humor reflection upon a grave, world-ending responsibility. Given that nuclear weapons were inevitable, I'm glad my country has some, that apparently MAD has worked thus far, and that there hasn't yet been an apocalyptic incident. I also hope that certain autocratic world leaders are taking their vitamins, and won't get even nuttier.)
- RCitronsBroker 3y ago
  considering how close to the edge of extermination we already have been brought, calling MAD successful leaves a real bad taste in my mouth.
  - neilv 3y ago
    I had left it at "apparently MAD has worked thus far", not getting into the almost didn't. It still might end us. Also, think of all the past plausible/likely bad scenarios that MAD averted, such as involving the Soviets. I sometimes joke about a utopian future of humanity, all peace and love, for thousands of years... until we encounter malevolent space aliens. Humanity had lost the ability to even reason about such a conflict, unable to imagine many options. Fortunately, humanity had maintained the cryogenic suspension of Kissinger and other historical figures capable of cold-blooded ruthlessness. And there was a Thaw button. (In some versions of the story, as soon as the anti-dream team wakes up and starts talking, we decide they're even worse than the space aliens.)
    - kwhitefoot 3y ago
      Not quite the same but you have reminded me of Clarke's The City and The Stars.
    - flir 3y ago
      This is the plot to Demolition Man :D
    - whatshisface 3y ago
      As a compromise, I propose that we get rid of Putin et al. but keep Paradox Interactive games.
- gumby 3y ago
  > In front of that Draper building, there was a strange pillar, which I liked to imagine housed a geo-positioning reference point -- perhaps even the origin of a coordinate system. Then, if ever there were an ICBM guidance failure, on that fateful day, Draper might be ground (0,0,0). Did you ever examine it closely? Nobody would try to stop you. It had a metal cap with a hole in it and some scratches. A few times over the years I remember seeing a tent around it. On those cases I didn't try to see if anyone would try to stop me! I know I and my friends thought the same as you.
  - neilv 3y ago
    I'm glad I'm not the only one with an overactive imagination. :) Yep, I took a look at it and photographed it a few times over the years. I never saw a tent, nor any attention given to it by anyone other than me. If it didn't have that cap on it, I might've leaned a bit more towards suspecting it being vestigial from a lamp or flagpole. IIRC, the pillar was still there after the snazzy remodeling of the building entrance that happened in fairly recent years.
- petermcneeley 3y ago
  MAD not only has prevented nuclear war but prevented many conventional wars. MAD is the worlds most successful peace initiatives. MAD is a defensive fortification in a world were we thought such was impossible. Those with weak stomachs, the Weizenbaums of this world, are too myopic to realize this truth.
RCitronsBroker 3y ago
this paper, without failing, makes me cry.
neilv 3y ago
A couple decades ago, when I was talking with the inventor of some aviation safety stuff, about working with them, he said some of their work involves the military, and asked whether I'd be OK with that. I (thinking of some of the better ideals) said something favorable about the sense of honor/duty/etc., and that I just didn't work on weapons systems nor domestic surveillance. I'd actually thought of it before, and those two spaces seemed problematic. Earlier, I'd actually worked (on non-mil things) with someone who at previous company had been a manager for a defensive military system. Which system everyone had seen on CNN, shooting down conventional missiles that some aggressor was launching at civilians. Which was great (albeit, CNN sometimes strayed into propaganda, and maybe not all the great stories were true). But, thinking of that great defensive use, I also thought that I wouldn't want to someday see on CNN something I worked on being used for something terrible. It wouldn't be sufficient to claim that I thought the Slaughtertron 3000 would only ever be used for good. Even a purely defensive system (Shieldtron 4000) could be used to shield an aggressor -- leaving the victims less able to defend themselves, and making aggression more viable. We'd all do well to read things like this article, and think about what we choose to work on, and how we do it. Then follow through on our better thoughts, in our behavior. In particular, the tech industry overall has become a festival of sociopathy. With many techbros, in moments of reflection, rationalizing, "but I gotta feed my family", while pulling down quite a lot more money than we need to do that. That wasn't very defensible when coders were regularly hopping jobs with 5-6 figure pay bumps each time, or doing startups that essentially involved selling out users and oftentimes later investors, ignoring regulations, etc. Even now that jobs are looking a lot less certain for most people, we still need to be very skeptical of ourselves every time we grasp for rationale.
- kolinko 3y ago
  Without systems like this (patriot and the others) people in Ukraine would be without heat and electricity last winter and this winter. I don’t know if you followed the stats - I did (it’s a neigboring country to mine and I have friends there), and it was dosens of rockets&drones a day targetting civilian infrastructure right in the middle of winter. Since patriot and other defence systems were set up, over 90% iirc attacks were repelled.
d0mine 3y ago
He described the future 40 years ahead of his time. There is even no need for a pilot today. Videos of drones killing people are freely available on reddit.
- kolinko 3y ago
  It’s people operating the drones, there is a human in the loop. Although it’s more because of a lack of proper guidance ai than the lack of will. Even with semi-autonomous Switchblades there is always a human choosing the initial targets - thanks to the voice in the post and other like this military has agreed that fully autonomous systems are something we should approach very carefuly and only under very specific scenarios.
uwagar 3y ago
enlightening. thanks for posting. i suspect he'd have gotten canceled if he wrrote this today.

64 points by drcwpl 3y ago

This has been floating around HN for two days and hasn't received any comments, but it's an insanely prescient paper much apropos the 1800+ comments in threads on "video AI".

ebcode 3y ago

Thanks for posting this. For those unaware, Weizenbaum wrote one of the first chatbots, ELIZA. https://en.wikipedia.org/wiki/Joseph_Weizenbaum https://en.wikipedia.org/wiki/Joseph_Weizenbaum

dyoo1979 3y ago

Thank you for bringing this up. I had not read this before, but it speaks to my deep concerns.

neilv 3y ago

> [...] the building next to the one in which I work, the world famous Draper Laboratory. [...] devoted almost entirely to missile guidance and submarine navigation. In front of that Draper building, there was a strange pillar, which I liked to imagine housed a geo-positioning reference point -- perhaps even the origin of a coordinate system. Then, if ever there were an ICBM guidance failure, on that fateful day, Draper might be ground (0,0,0). Like an engineer standing under the bridge they built, as the first trucks drive over it. I was thinking of that as intentional/conscious by the engineers. (BTW, I'm not criticizing; only a little dark humor reflection upon a grave, world-ending responsibility. Given that nuclear weapons were inevitable, I'm glad my country has some, that apparently MAD has worked thus far, and that there hasn't yet been an apocalyptic incident. I also hope that certain autocratic world leaders are taking their vitamins, and won't get even nuttier.)

RCitronsBroker 3y ago
considering how close to the edge of extermination we already have been brought, calling MAD successful leaves a real bad taste in my mouth.
- neilv 3y ago
  I had left it at "apparently MAD has worked thus far", not getting into the almost didn't. It still might end us. Also, think of all the past plausible/likely bad scenarios that MAD averted, such as involving the Soviets. I sometimes joke about a utopian future of humanity, all peace and love, for thousands of years... until we encounter malevolent space aliens. Humanity had lost the ability to even reason about such a conflict, unable to imagine many options. Fortunately, humanity had maintained the cryogenic suspension of Kissinger and other historical figures capable of cold-blooded ruthlessness. And there was a Thaw button. (In some versions of the story, as soon as the anti-dream team wakes up and starts talking, we decide they're even worse than the space aliens.)
  - kwhitefoot 3y ago
    Not quite the same but you have reminded me of Clarke's The City and The Stars.
  - flir 3y ago
    This is the plot to Demolition Man :D
  - whatshisface 3y ago
    As a compromise, I propose that we get rid of Putin et al. but keep Paradox Interactive games.
gumby 3y ago
> In front of that Draper building, there was a strange pillar, which I liked to imagine housed a geo-positioning reference point -- perhaps even the origin of a coordinate system. Then, if ever there were an ICBM guidance failure, on that fateful day, Draper might be ground (0,0,0). Did you ever examine it closely? Nobody would try to stop you. It had a metal cap with a hole in it and some scratches. A few times over the years I remember seeing a tent around it. On those cases I didn't try to see if anyone would try to stop me! I know I and my friends thought the same as you.
- neilv 3y ago
  I'm glad I'm not the only one with an overactive imagination. :) Yep, I took a look at it and photographed it a few times over the years. I never saw a tent, nor any attention given to it by anyone other than me. If it didn't have that cap on it, I might've leaned a bit more towards suspecting it being vestigial from a lamp or flagpole. IIRC, the pillar was still there after the snazzy remodeling of the building entrance that happened in fairly recent years.
petermcneeley 3y ago
MAD not only has prevented nuclear war but prevented many conventional wars. MAD is the worlds most successful peace initiatives. MAD is a defensive fortification in a world were we thought such was impossible. Those with weak stomachs, the Weizenbaums of this world, are too myopic to realize this truth.

RCitronsBroker 3y ago

this paper, without failing, makes me cry.

neilv 3y ago

A couple decades ago, when I was talking with the inventor of some aviation safety stuff, about working with them, he said some of their work involves the military, and asked whether I'd be OK with that. I (thinking of some of the better ideals) said something favorable about the sense of honor/duty/etc., and that I just didn't work on weapons systems nor domestic surveillance. I'd actually thought of it before, and those two spaces seemed problematic. Earlier, I'd actually worked (on non-mil things) with someone who at previous company had been a manager for a defensive military system. Which system everyone had seen on CNN, shooting down conventional missiles that some aggressor was launching at civilians. Which was great (albeit, CNN sometimes strayed into propaganda, and maybe not all the great stories were true). But, thinking of that great defensive use, I also thought that I wouldn't want to someday see on CNN something I worked on being used for something terrible. It wouldn't be sufficient to claim that I thought the Slaughtertron 3000 would only ever be used for good. Even a purely defensive system (Shieldtron 4000) could be used to shield an aggressor -- leaving the victims less able to defend themselves, and making aggression more viable. We'd all do well to read things like this article, and think about what we choose to work on, and how we do it. Then follow through on our better thoughts, in our behavior. In particular, the tech industry overall has become a festival of sociopathy. With many techbros, in moments of reflection, rationalizing, "but I gotta feed my family", while pulling down quite a lot more money than we need to do that. That wasn't very defensible when coders were regularly hopping jobs with 5-6 figure pay bumps each time, or doing startups that essentially involved selling out users and oftentimes later investors, ignoring regulations, etc. Even now that jobs are looking a lot less certain for most people, we still need to be very skeptical of ourselves every time we grasp for rationale.

kolinko 3y ago
Without systems like this (patriot and the others) people in Ukraine would be without heat and electricity last winter and this winter. I don’t know if you followed the stats - I did (it’s a neigboring country to mine and I have friends there), and it was dosens of rockets&drones a day targetting civilian infrastructure right in the middle of winter. Since patriot and other defence systems were set up, over 90% iirc attacks were repelled.

d0mine 3y ago

He described the future 40 years ahead of his time. There is even no need for a pilot today. Videos of drones killing people are freely available on reddit.

kolinko 3y ago
It’s people operating the drones, there is a human in the loop. Although it’s more because of a lack of proper guidance ai than the lack of will. Even with semi-autonomous Switchblades there is always a human choosing the initial targets - thanks to the voice in the post and other like this military has agreed that fully autonomous systems are something we should approach very carefuly and only under very specific scenarios.

uwagar 3y ago

enlightening. thanks for posting. i suspect he'd have gotten canceled if he wrrote this today.