7 ms·

Why do we tell ourselves scary stories about AI?

59 points by lschueller 6mo ago

Zigurd 6mo ago
If I can plausibly say I'm making something super dangerous, the government is likely to want to be the first government to have it. If the check clears before they figure out if I'm BSing them or not, it's a win.
afavour 6mo ago
It does feel like a bizarre moment, where the AI companies are deliberately trying to scare us about their own product in a bid to, I think, show the inevitability of it? Or to sell themselves as the one responsible power to constrain it? It's very odd. "It's going to take all your jobs" is not a great selling point to the everyday public.
- mememememememo 6mo ago
  People are renowned voting or buying in their own worst interests. And it goes back to before Trump and in many countries.
  - afavour 6mo ago
    Yes but in order to get someone to vote against their interests you need to sell them on something else that's a benefit. They don't just automatically vote against themselves. "This technology might escape our control, might devastate the economy but also serves as a serviceable chatbot for your entertainment" isn't a vote winner.
    - mememememememo 6mo ago
      The way out of this for the seller is to lie and conflate the rich with everyone. "This technology will make your retirement account grow and other countries will be giving us money!"
- lschueller 6mo ago
  It is very odd, indeed. It's a bit of both well known "hells of marketing": fomo on the one side (you better use us as heavily as possible), combined with mysticism of "we don't know what we created, but it's powerful and you better follow us to be on the right side"
- bpodgursky 6mo ago
  They are being honest and you don't want to deal with the implications, so you stretch for conspiracy theories. The ones at the top are the true believers. Engage with them at that level.
  - 0x4e 6mo ago
    That's a good view point. Perhaps they're not being alarmists or trying to scare people, but being honest about the capabilities. Perhaps it can be better articulated and framed in a way that's well received. But, maybe that would be over-promising or not being honest about the future.
- MisterTea 6mo ago
  > It does feel like a bizarre moment, where the AI companies are deliberately trying to scare us about their own product That is direct CEO to CEO marketing. They're working really hard to convince high up decision makers that these tools will lower their head count and reduce costs.
- unethical_ban 6mo ago
  They're selling the product to the class of people who would love for it to take the jobs of our class of people.
- xg15 6mo ago
  Yeah, the messaging felt weirdly pyromanic, like telling everyone about the unimaginable dangers of fire and then saying that's why I have I to burn everything, to protect us from the fire...
- deepsquirrelnet 6mo ago
  There are so many reason if you look at how it's being sold. * We need to completely deregulate these US companies so China doesn't win and take us over * We need to heavily regulate anybody who is not following the rules that make us the de-facto winner * This is so powerful it will take all the jobs (and therefore if you lead a company that isn't using AI, you will soon be obsolete) * If you don't use AI, you will not be able to function in a future job * We need to lineup an excuse to call our friends in government and turn off the open source spigot when the time is right They have chosen fear as a motivator, and it is clearly working very well. It's easier to use fear now, while it's new and then flip the narrative once people are more familiar with it than to go the other direction. Companies are not just telling a story to hype their product, but why they alone are the ones that should be entrusted to build it.
mememememememo 6mo ago
I read and experience scary stories about AI already. It is not a future maybe thing.
zaps 6mo ago
Why do we tell ourselves scary stories about anything?
5asaKI 6mo ago
Indeed. Apart from the obvious prompt research frauds mentioned in the article, the model learned all deceptive behaviors from hundreds of Yudkowsky scenarios that are easily available. It literally plagiarizes its supposed free will like a good IP laundromat.
chrisbrandow 6mo ago
I don’t think the fact that the robot was instructed to lie to a human and was able to do so successfully makes the story much less scary for most people.
nalekberov 6mo ago
> “The last four years have demonstrated that AI agents can acquire the will to survive and that AIs have already learned how to lie.” Why Harari feels an obligation to comment about everything is of course beyond me, but describing 'AI' as if it takes independent decisions to lie, make moral judgements, etc. demonstrates either that he has zero clue how 'AI' trains itself or that he chooses to mislead the audience.
- 0x4e 6mo ago
  Agreed. But, could it be trained ti be deceiving? Especially when we bake-in advertising into it?
- yoz-y 6mo ago
  Isn't the problem precisely that it does not take moral judgements? My opinion on all of this is constantly shifting, but right now my main issue is that-like self driving-it seems 90-95% correct and 5-10% catastrophically wrong. Due to the sheer speed and volume of output it produces I have grown complacent and exhausted, so when I give it simple tasks I assume it is correct and then is the time when "it deletes" all of your files.
Forgeties79 6mo ago
LLM companies’ behavior, AI evangelists, and the investment fervor around it all, are telling us the scary stories.
genie3io 6mo ago
[dead]
0x4e 6mo ago
Because we don't like uncertainty, and the AI future is uncertain. There are multiple high probability scenarios. Because we're seeing how its capabilities increase overtime. I find the rate at which I prefer to go to an AI than an UpWorker is scary. Because we——the people——are not in control of it. We're at the whims of whatever it and the tech bros want (technocracy).
- dryarzeg 6mo ago
  > technocracy What you mentioned is not a technocracy. Technocracy is when all decisions are made by real specialists in the field, based on scientific methods (simply speaking). What you mentioned is a plutocracy, a form of oligarchy in which decisions are made by people of great wealth. I couldn’t just ignore this because, in my view, technocracy (as I’ve described it) still has some merit - for instance, appointing only genuine economists to head a hypothetical Ministry of Economy makes some sense - whereas oligarchy and plutocracy have nothing good to offer. Of course, this is just my personal opinion.
  - SpicyLemonZest 6mo ago
    I think most AI execs I'm familiar with would, if they were the god-monarch of humanity, recruit real specialists applying scientific methods to make most decisions. They seem like the kind of people who would understand that the Ministry of Economy is doing valuable things which shouldn't be compromised for personal expediency. Does that really make it any better?
    - dryarzeg 6mo ago
      > if they were the god-monarch of humanity In that case, we're not talking about an oligarchy or a technocracy either. What you have described is an autocracy - a rule by one. When there's some kind of "god-monarch", the people heading the Ministry of Economy will be controlled by this "god-monarch" and it's unclear if this can be called technocracy or not (at least it is unclear to me; maybe I'm stupid, who knows). > Does that really make it any better? Honestly? If you're asking about "would it be any better than now" - I'm not really sure, because I'm not in power to access the actions of the people who hold the positions equivalent to the head of Ministry of Economy - the economy is not my field, I'm not a specialist here. I would only point to the example I'm familiar with (and you're probably not; I'm sorry, I just couldn't think about something like this that I can verify) - in Ukraine, there's a "Ministry of Digital Transformation". This ministry was headed by a Mykhailo Fedorov, whose primary and as far as I know, he studied at the "Faculty of Sociology and Management". Well, that’s not the main point, as he’s studied elsewhere too. The problem lies elsewhere. His decisions have been criticized on more than one occasion by genuine experts - for example, the project known as "Diya", or "the state in a smartphone"; in short, it’s something like access to documents and various government services all in one app. It’s a long story… In short, as a result, there were (presumably) data leaks, and the service crashed more than once or twice due to its flawed security, and all sorts of problems were found with it - you name it, it had it. It's such a shame, to be honest... You can't just go and play with things like that. And now that person is serving as a... head of Ministry of Defence. Hell. To add insult to the injury, guess who is now taking his place at a Ministry of Digital Transformation? Oleksandr Bornyakov, who, as far as I know, holds a degree in marketing. Marketing. ...Nice. Well maybe I don't know something, who knows, who knows... but the decisions, or, rather, their consequences seem to be... let's settle with "terrible". I'm pretty sure you can recall some similar examples yourself. My point is that although the scenario you described is not as good - because, I guess, no one really wants "god-monarch" controlling (although not making directly) all of the decisions. But if our hypothetical Ministry of the Economy were run by genuine experts who, moreover, work for the good of society - or at least the state as a whole - rather than just lining their own pockets, well, that sounds better than an idiocracy. That was my point.
Rzor 6mo ago
For regulatory capture, of course. They are not fooling me. There may be other motives, and the more ever-doom-looking crowd can find something in it for themselves as well, but you don't have to dig any deeper if you are looking for an explanation for the perspective of the people actually building it. The Chinese tech sector popularizing cheap and open source models sure did a number on that narrative, too. Llama models, a while ago, too.
ramon156 6mo ago
I feel like this article is more written towards non-techies. A decent amount of programmers have touched coding agents, and know it "kind of" does it's job. It's good enough for some tasks... I cannot be arsed to figure out how to edit a graph in Drupal, so I ask Claude. Claude fixes it, and it's not anymore broken than it already was. Win win. However, that's where I stop my agent usage. I let ~~Claude~~ GLM do the following: - Fix tedious tasks that cost me more to figure out than I care for - Research something I'm not familiar with, and give me the facts it had found, and even then I end up looking at the source myself
- MrDarcy 6mo ago
  This usage pattern is a few months behind the curve. It’s effective at full on feature development now. Keep it fed with plans and it’ll keep implementing, leaving the codebase better than it found it each cycle.
everdrive 6mo ago
One thing that strikes me that I never really see anyone discuss is that we've been afraid of conscious computers for a _long_ time. Back in the 50s and before people were quite afraid that we'd build conscious computers. This was long before there was any sense that could actually accomplish the task. I think that similarly to seeing faces in the clouds, we imagine a consciousness where none exists. (eg: a rain god rather than a complex system of physics and chemistry) Even LLMs, which blow past any normal Turing test methods, are still not conscious. But they certainly _feel_ conscious. They trigger the same intuitions that we rely on for consciousness. You ask yourself "how would I need to frame this question so that Claude would understand it?" You use the same mental hardware that you'd use for consciousness. So, you have an historical and permanent fear of consciousness in a powerful entity where no consciousness actually exists combined with the fact that we created things which definitely seem conscious. (not to mention that consciousness could genuinely be on its way soon)
- ACCount37 6mo ago
  Are they? Not conscious? If you list out every prominent theory of consciousness, you'd find that about a quarter rules out LLMs, a quarter tentatively rules LLMs in, and what remains is "uncertain about LLMs". And, of course, we don't know which theory of consciousness is correct - or if any of them is. So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't?
  - jacquesm 6mo ago
    The fact that it's a box with a plug and a state that can be fully known. A conscious entity has a state that can not be fully known. Far smarter people than me have made this argument and in a much more eloquent way. Turing aimed too low.
    - saHqtr 6mo ago
      And the chatbots don't even pass the Turing test. I've never had a normal conversation. It's always prompt => lengthy, cocksure and somewhat autistic response. They are very easily distinguishable.
crancher 6mo ago
We're hardwired to fear the rustle in the grass, and successful infrastructure gets backgrounded. Pain is a signal. How much time do you spend contemplating your skeleton outside of pain-related skeletal events?
ACCount37 6mo ago
It's simple. It's because AI is the scariest technology ever made. Human intelligence has proven itself capable of doing a lot of scary things. And AI research is keen on building ever more capable and more scalable intelligence. By now, the list of advantages humans still have over machines is both finite and rapidly diminishing. If that doesn't make you uneasy, you're in denial.
- causal 6mo ago
  "Why be afraid of nukes it's not like they WANT to blow up"
- sublinear 6mo ago
  This is untrue. What is being diminished is the value of humans doing repetitive or uncreative tasks. Many have built their careers from that kind of work in the past and yes they are threatened, but that kind of work is inherently not collaborative and more vocational.
  - bauerd 6mo ago
    The vast majority of people on this planet work repetitive, uncreative jobs.
    - sublinear 6mo ago
      There is no such job done by humans today that is 100% uncreative, but people will continue to insist there is. The devaluing may come from AI pressure, but the harm is coming from humans foolishly not seeing the value in what's left behind. Most people have not and will not lose their jobs.
    - jaredcwhite 6mo ago
      Oh? And what extensive knowledge and experience makes YOU qualified to determine what "the vast majority of people on this planet" are doing for work and if those tasks are creative or uncreative?
bharat1010 6mo ago
The point about AI companies actively hyping the danger of their own products is something I hadn't really thought about before — it's a strange kind of marketing when you think about it.
ggambetta 6mo ago
We tell ourselves scary stories about everything new. Advances in electricity + medicine == FRANKENSTEIN!
vdelpuerto 6mo ago
[flagged]
SpicyLemonZest 6mo ago
The actual contents of this article are making reasonable arguments I largely agree with. It would be very surprising for LLM-based AI systems to act as monomanaical goal optimizers, since they're trained on human text and humans are extremely bad at goal-oriented behavior. (My goals for today include a number of work and self-maintenance tasks, and the time I'm spending here writing out a HN comment does not all help achieve them - I suspect most people reading this comment are in the same boat.) It's very frustrating that the magazine wrote such a dumb headline which guarantees people won't talk about the issues the article raised. Obviously non-goal-oriented systems can still have important negative effects.
- mitthrowaway2 6mo ago
  I feel like if it took thirty or forty more years of research before a more advanced, more goal-oriented AI became capable of destroying humanity for paperclips, that would still be a bad outcome. The road ahead is long and we're moving too fast to only consider what's in the rear view mirror.
GolfPopper 6mo ago
Why does the uncanny valley[1] exist? (If it truly does.) What in our evolutionary history gave us a reflexive rejection of things that seem human but aren't? 1. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley
- Eldt 6mo ago
  My guess? Psychological disorders, like psychosis
- zbikowski 6mo ago
  I always imagined this to have evolved from a long history of humans getting sick around rotting corpses. The logical move is to stay away from them, and thinking they're freaky-looking is a good driver for that. Though the idea of neandertals eliciting a similar reaction has always been interesting to me.
- benj111 6mo ago
  Interesting question. Perhaps it's down to the fact that a lot of communication is contextual/ group bonding? You are expected to behave a certain way. If you laugh at something the group doesn't find funny, you are excluded from the group. Visa versa. If something does behave quite right, it doesn't belong in the group? I'm autistic for what it's worth. I'm not sure if that gives me better of worse insight into group dynamics.
throwaway613746 6mo ago
[dead]
KaoruAoiShiho 6mo ago
TLDR: Writer hasn't heard of agents.
jacquesm 6mo ago
This article would be a lot more digestible if we didn't have actual scary data rather than just stories. Not a day goes by without some prompt injection oopsie, security gotcha, deepfake or some sandbox escape artist demonstration and tbh I'm impressed but more to the point where I don't doubt this is dangerous tech, I'm sure of it. This is roughly 1995 again and we're going to find out all over why mixing instructions and data was a spectacularly bad idea. Only now with human language as the input stream, which is far more expressive than HTML or SQL ever were. So now everybody is a hacker. At least in that sense it has leveled the playing field I guess.
FatherOfCurses 6mo ago
We tell ourselves scary stories about AI because humanity is rife with stories where a new idea or technology has had unintended negative consequences. AI bros just care about selling their product to another company and cashing out, they have absolutely no regard for their legacy.
- cbogie 6mo ago
  we tell ourselves scary stories because that’s what we do with language. and now our language isn’t just ours anymore.
the_af 6mo ago
I think the most insightful bit is buried in the article: > Perhaps because this is the best advertising money can’t buy. People like Harari and others repeat these accounts like ghost stories around a campfire. The public, awed and afraid, marvels at the capabilities of AI. And that's mostly it. PR. Publicity. Fear is good publicity if it emphasizes AI's capabilities. And people like Harari (or Gladwell) tell interesting and awe-inspiring stories that do not necessarily have much rigor or fact-checking in them. They simplify for storytelling purposes, which can result in misleading stories. I am worried about AI, but not about superintelligent AI that will exterminate or enslave us. I'm worried about AI as a tool to concentrate wealth and power in the hands of the current amoral entrepreneurial elite. I'm not sure whether I trust ChatGPT, but I sure as hell do NOT trust Sam Altman et al. Or, in other words, I subscribe to Ted Chiang's very apt remark about what we really fear: > “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker, “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”
Swoerd 6mo ago
[dead]
netdevphoenix 6mo ago
There is a very interesting book that explores the West's generally negative view of artificial intelligence whenever it is portrayed in media (Skynet) while Japanese media tends to have a positive view (Astro Boy).
yanis_t 6mo ago
I wish we didn't call this AI as the term is crazily overloaded. Those are programs. The only difference is how we write them. Not with "if"s and "for"s. We take a bunch of bits that do nothing. Then we organize them in a way so that it outputs whatever it is we want.
- mitthrowaway2 6mo ago
  What kind of AI do you envision that would ever not fit that description?
RivieraKid 6mo ago
I 100% agree with this take, I find AI completely non-scary, especially in the sense that it's some kind of a conscious entity that will want to take over. I find these people almost delusional. It's a powerful tool, so it can be dangerous if used by people with bad intentions, so there's some real danger here, but my intuition is that it will be fine. The ratio between the power of people with good vs bad intentions shouldn't change too dramatically. The only scary part is that it could be bad for my future as a software developer. That said, I think it will be net benefit for the average worker - the average person will work less and earn more.
dclowd9901 6mo ago
My favorite part of this article was this bit, and naturally so, since I love the author: > Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker(opens a new tab), “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.” I didn't realize it til I read it here, but yes, my fear isn't really about the machine, it's about the machine that drives the machine. We already have a class of amoral beings that treat the world as an expendable thing and are willing to burn it down for profit. We should focus on getting rid of that problem first.
greton7 6mo ago
[dead]
latentsea 6mo ago
Humans are the existence proof we seem to be chasing and the goal appears to be to hit a superset of humanities collective capabilities. I don't know if you've seen what we do to each other but... I'm afraid of a decent chunk of us too. If what we build is a superset of our capabilities then I am afraid of it, because I am already afraid of us.
garte 6mo ago
Great article, most of all the latter parts about what autopoiesis means regarding computers.
benj111 6mo ago
People were worried about trains in tunnels 180 years ago. People worry about new things. See also: the internet, video games, mobile phones, various genres of music. http://www.boxpeopleandplaces.co.uk/building-box-tunnel.html http://www.boxpeopleandplaces.co.uk/building-box-tunnel.html
ladax72707 6mo ago
> It occurred to me that if I could invent a machine - a gun - which could by its rapidity of fire, enable one man to do as much battle duty as a hundred, that it would, to a large extent supersede the necessity of large armies, and consequently, exposure to battle and disease [would] be greatly diminished.
xemdetia 6mo ago
The only thing I find scary about the current AI is how many AI companies became completely untethered from basic ethical controls openly. There is no facade of decency and it seems like a lot of them are running from their own shadow. I feel like there is a middle ground that could have been taken to bring content experts in instead of looting the web the world has built.
tim333 6mo ago
>After talking to experts, I was convinced there’s no reason to fear AIs developing a will to live, and then tricking or destroying us to avoid shutdown and take over the world. Unless, of course, we tell them to. Once we have super intelligent AIs I give it a day or two till someone tries that, prompting something like take over the world for me.
hateful 6mo ago
This may be pure conjecture, but my first thought is always that it might be a marketing ploy by the AI companies to make their products seem more advanced than they really are.
qgin 6mo ago
We’ve seen an incredibly powerful technology follow multiple exponential curves in its capability, but we’re supposed to ask why we’re telling ourselves “stories” if we think about what will happen if that technology continues to follow the curves it has been following without sign of hitting any walls? Is AGI certain? No. But there’s currently no specific reason to believe it isn’t coming in the next few years.

59 points by lschueller 6mo ago

If I can plausibly say I'm making something super dangerous, the government is likely to want to be the first government to have it. If the check clears before they figure out if I'm BSing them or not, it's a win.

afavour 6mo ago

It does feel like a bizarre moment, where the AI companies are deliberately trying to scare us about their own product in a bid to, I think, show the inevitability of it? Or to sell themselves as the one responsible power to constrain it? It's very odd. "It's going to take all your jobs" is not a great selling point to the everyday public.

mememememememo 6mo ago
People are renowned voting or buying in their own worst interests. And it goes back to before Trump and in many countries.
- afavour 6mo ago
  Yes but in order to get someone to vote against their interests you need to sell them on something else that's a benefit. They don't just automatically vote against themselves. "This technology might escape our control, might devastate the economy but also serves as a serviceable chatbot for your entertainment" isn't a vote winner.
  - mememememememo 6mo ago
    The way out of this for the seller is to lie and conflate the rich with everyone. "This technology will make your retirement account grow and other countries will be giving us money!"
lschueller 6mo ago
It is very odd, indeed. It's a bit of both well known "hells of marketing": fomo on the one side (you better use us as heavily as possible), combined with mysticism of "we don't know what we created, but it's powerful and you better follow us to be on the right side"
bpodgursky 6mo ago
They are being honest and you don't want to deal with the implications, so you stretch for conspiracy theories. The ones at the top are the true believers. Engage with them at that level.
- 0x4e 6mo ago
  That's a good view point. Perhaps they're not being alarmists or trying to scare people, but being honest about the capabilities. Perhaps it can be better articulated and framed in a way that's well received. But, maybe that would be over-promising or not being honest about the future.
MisterTea 6mo ago
> It does feel like a bizarre moment, where the AI companies are deliberately trying to scare us about their own product That is direct CEO to CEO marketing. They're working really hard to convince high up decision makers that these tools will lower their head count and reduce costs.
unethical_ban 6mo ago
They're selling the product to the class of people who would love for it to take the jobs of our class of people.
xg15 6mo ago
Yeah, the messaging felt weirdly pyromanic, like telling everyone about the unimaginable dangers of fire and then saying that's why I have I to burn everything, to protect us from the fire...
deepsquirrelnet 6mo ago
There are so many reason if you look at how it's being sold. * We need to completely deregulate these US companies so China doesn't win and take us over * We need to heavily regulate anybody who is not following the rules that make us the de-facto winner * This is so powerful it will take all the jobs (and therefore if you lead a company that isn't using AI, you will soon be obsolete) * If you don't use AI, you will not be able to function in a future job * We need to lineup an excuse to call our friends in government and turn off the open source spigot when the time is right They have chosen fear as a motivator, and it is clearly working very well. It's easier to use fear now, while it's new and then flip the narrative once people are more familiar with it than to go the other direction. Companies are not just telling a story to hype their product, but why they alone are the ones that should be entrusted to build it.

mememememememo 6mo ago

I read and experience scary stories about AI already. It is not a future maybe thing.

zaps 6mo ago

Why do we tell ourselves scary stories about anything?

5asaKI 6mo ago

Indeed. Apart from the obvious prompt research frauds mentioned in the article, the model learned all deceptive behaviors from hundreds of Yudkowsky scenarios that are easily available. It literally plagiarizes its supposed free will like a good IP laundromat.

chrisbrandow 6mo ago

I don’t think the fact that the robot was instructed to lie to a human and was able to do so successfully makes the story much less scary for most people.

nalekberov 6mo ago

> “The last four years have demonstrated that AI agents can acquire the will to survive and that AIs have already learned how to lie.” Why Harari feels an obligation to comment about everything is of course beyond me, but describing 'AI' as if it takes independent decisions to lie, make moral judgements, etc. demonstrates either that he has zero clue how 'AI' trains itself or that he chooses to mislead the audience.

0x4e 6mo ago
Agreed. But, could it be trained ti be deceiving? Especially when we bake-in advertising into it?
yoz-y 6mo ago
Isn't the problem precisely that it does not take moral judgements? My opinion on all of this is constantly shifting, but right now my main issue is that-like self driving-it seems 90-95% correct and 5-10% catastrophically wrong. Due to the sheer speed and volume of output it produces I have grown complacent and exhausted, so when I give it simple tasks I assume it is correct and then is the time when "it deletes" all of your files.

Forgeties79 6mo ago

LLM companies’ behavior, AI evangelists, and the investment fervor around it all, are telling us the scary stories.

genie3io 6mo ago

[dead]

0x4e 6mo ago

Because we don't like uncertainty, and the AI future is uncertain. There are multiple high probability scenarios. Because we're seeing how its capabilities increase overtime. I find the rate at which I prefer to go to an AI than an UpWorker is scary. Because we——the people——are not in control of it. We're at the whims of whatever it and the tech bros want (technocracy).

dryarzeg 6mo ago
> technocracy What you mentioned is not a technocracy. Technocracy is when all decisions are made by real specialists in the field, based on scientific methods (simply speaking). What you mentioned is a plutocracy, a form of oligarchy in which decisions are made by people of great wealth. I couldn’t just ignore this because, in my view, technocracy (as I’ve described it) still has some merit - for instance, appointing only genuine economists to head a hypothetical Ministry of Economy makes some sense - whereas oligarchy and plutocracy have nothing good to offer. Of course, this is just my personal opinion.
- SpicyLemonZest 6mo ago
  I think most AI execs I'm familiar with would, if they were the god-monarch of humanity, recruit real specialists applying scientific methods to make most decisions. They seem like the kind of people who would understand that the Ministry of Economy is doing valuable things which shouldn't be compromised for personal expediency. Does that really make it any better?
  - dryarzeg 6mo ago
    > if they were the god-monarch of humanity In that case, we're not talking about an oligarchy or a technocracy either. What you have described is an autocracy - a rule by one. When there's some kind of "god-monarch", the people heading the Ministry of Economy will be controlled by this "god-monarch" and it's unclear if this can be called technocracy or not (at least it is unclear to me; maybe I'm stupid, who knows). > Does that really make it any better? Honestly? If you're asking about "would it be any better than now" - I'm not really sure, because I'm not in power to access the actions of the people who hold the positions equivalent to the head of Ministry of Economy - the economy is not my field, I'm not a specialist here. I would only point to the example I'm familiar with (and you're probably not; I'm sorry, I just couldn't think about something like this that I can verify) - in Ukraine, there's a "Ministry of Digital Transformation". This ministry was headed by a Mykhailo Fedorov, whose primary and as far as I know, he studied at the "Faculty of Sociology and Management". Well, that’s not the main point, as he’s studied elsewhere too. The problem lies elsewhere. His decisions have been criticized on more than one occasion by genuine experts - for example, the project known as "Diya", or "the state in a smartphone"; in short, it’s something like access to documents and various government services all in one app. It’s a long story… In short, as a result, there were (presumably) data leaks, and the service crashed more than once or twice due to its flawed security, and all sorts of problems were found with it - you name it, it had it. It's such a shame, to be honest... You can't just go and play with things like that. And now that person is serving as a... head of Ministry of Defence. Hell. To add insult to the injury, guess who is now taking his place at a Ministry of Digital Transformation? Oleksandr Bornyakov, who, as far as I know, holds a degree in marketing. Marketing. ...Nice. Well maybe I don't know something, who knows, who knows... but the decisions, or, rather, their consequences seem to be... let's settle with "terrible". I'm pretty sure you can recall some similar examples yourself. My point is that although the scenario you described is not as good - because, I guess, no one really wants "god-monarch" controlling (although not making directly) all of the decisions. But if our hypothetical Ministry of the Economy were run by genuine experts who, moreover, work for the good of society - or at least the state as a whole - rather than just lining their own pockets, well, that sounds better than an idiocracy. That was my point.

Rzor 6mo ago

For regulatory capture, of course. They are not fooling me. There may be other motives, and the more ever-doom-looking crowd can find something in it for themselves as well, but you don't have to dig any deeper if you are looking for an explanation for the perspective of the people actually building it. The Chinese tech sector popularizing cheap and open source models sure did a number on that narrative, too. Llama models, a while ago, too.

ramon156 6mo ago

I feel like this article is more written towards non-techies. A decent amount of programmers have touched coding agents, and know it "kind of" does it's job. It's good enough for some tasks... I cannot be arsed to figure out how to edit a graph in Drupal, so I ask Claude. Claude fixes it, and it's not anymore broken than it already was. Win win. However, that's where I stop my agent usage. I let ~~Claude~~ GLM do the following: - Fix tedious tasks that cost me more to figure out than I care for - Research something I'm not familiar with, and give me the facts it had found, and even then I end up looking at the source myself

MrDarcy 6mo ago
This usage pattern is a few months behind the curve. It’s effective at full on feature development now. Keep it fed with plans and it’ll keep implementing, leaving the codebase better than it found it each cycle.

everdrive 6mo ago

One thing that strikes me that I never really see anyone discuss is that we've been afraid of conscious computers for a _long_ time. Back in the 50s and before people were quite afraid that we'd build conscious computers. This was long before there was any sense that could actually accomplish the task. I think that similarly to seeing faces in the clouds, we imagine a consciousness where none exists. (eg: a rain god rather than a complex system of physics and chemistry) Even LLMs, which blow past any normal Turing test methods, are still not conscious. But they certainly _feel_ conscious. They trigger the same intuitions that we rely on for consciousness. You ask yourself "how would I need to frame this question so that Claude would understand it?" You use the same mental hardware that you'd use for consciousness. So, you have an historical and permanent fear of consciousness in a powerful entity where no consciousness actually exists combined with the fact that we created things which definitely seem conscious. (not to mention that consciousness could genuinely be on its way soon)

ACCount37 6mo ago
Are they? Not conscious? If you list out every prominent theory of consciousness, you'd find that about a quarter rules out LLMs, a quarter tentatively rules LLMs in, and what remains is "uncertain about LLMs". And, of course, we don't know which theory of consciousness is correct - or if any of them is. So, what is it that makes you so sure, oh so very certain, that LLMs just "feel" conscious but aren't?
- jacquesm 6mo ago
  The fact that it's a box with a plug and a state that can be fully known. A conscious entity has a state that can not be fully known. Far smarter people than me have made this argument and in a much more eloquent way. Turing aimed too low.
  - saHqtr 6mo ago
    And the chatbots don't even pass the Turing test. I've never had a normal conversation. It's always prompt => lengthy, cocksure and somewhat autistic response. They are very easily distinguishable.

crancher 6mo ago

We're hardwired to fear the rustle in the grass, and successful infrastructure gets backgrounded. Pain is a signal. How much time do you spend contemplating your skeleton outside of pain-related skeletal events?

ACCount37 6mo ago

It's simple. It's because AI is the scariest technology ever made. Human intelligence has proven itself capable of doing a lot of scary things. And AI research is keen on building ever more capable and more scalable intelligence. By now, the list of advantages humans still have over machines is both finite and rapidly diminishing. If that doesn't make you uneasy, you're in denial.

causal 6mo ago
"Why be afraid of nukes it's not like they WANT to blow up"
sublinear 6mo ago
This is untrue. What is being diminished is the value of humans doing repetitive or uncreative tasks. Many have built their careers from that kind of work in the past and yes they are threatened, but that kind of work is inherently not collaborative and more vocational.
- bauerd 6mo ago
  The vast majority of people on this planet work repetitive, uncreative jobs.
  - sublinear 6mo ago
    There is no such job done by humans today that is 100% uncreative, but people will continue to insist there is. The devaluing may come from AI pressure, but the harm is coming from humans foolishly not seeing the value in what's left behind. Most people have not and will not lose their jobs.
  - jaredcwhite 6mo ago
    Oh? And what extensive knowledge and experience makes YOU qualified to determine what "the vast majority of people on this planet" are doing for work and if those tasks are creative or uncreative?

bharat1010 6mo ago

The point about AI companies actively hyping the danger of their own products is something I hadn't really thought about before — it's a strange kind of marketing when you think about it.

ggambetta 6mo ago

We tell ourselves scary stories about everything new. Advances in electricity + medicine == FRANKENSTEIN!

vdelpuerto 6mo ago

[flagged]

SpicyLemonZest 6mo ago

The actual contents of this article are making reasonable arguments I largely agree with. It would be very surprising for LLM-based AI systems to act as monomanaical goal optimizers, since they're trained on human text and humans are extremely bad at goal-oriented behavior. (My goals for today include a number of work and self-maintenance tasks, and the time I'm spending here writing out a HN comment does not all help achieve them - I suspect most people reading this comment are in the same boat.) It's very frustrating that the magazine wrote such a dumb headline which guarantees people won't talk about the issues the article raised. Obviously non-goal-oriented systems can still have important negative effects.

mitthrowaway2 6mo ago
I feel like if it took thirty or forty more years of research before a more advanced, more goal-oriented AI became capable of destroying humanity for paperclips, that would still be a bad outcome. The road ahead is long and we're moving too fast to only consider what's in the rear view mirror.

GolfPopper 6mo ago

Why does the uncanny valley[1] exist? (If it truly does.) What in our evolutionary history gave us a reflexive rejection of things that seem human but aren't? 1. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley

Eldt 6mo ago
My guess? Psychological disorders, like psychosis
zbikowski 6mo ago
I always imagined this to have evolved from a long history of humans getting sick around rotting corpses. The logical move is to stay away from them, and thinking they're freaky-looking is a good driver for that. Though the idea of neandertals eliciting a similar reaction has always been interesting to me.
benj111 6mo ago
Interesting question. Perhaps it's down to the fact that a lot of communication is contextual/ group bonding? You are expected to behave a certain way. If you laugh at something the group doesn't find funny, you are excluded from the group. Visa versa. If something does behave quite right, it doesn't belong in the group? I'm autistic for what it's worth. I'm not sure if that gives me better of worse insight into group dynamics.

throwaway613746 6mo ago

[dead]

KaoruAoiShiho 6mo ago

TLDR: Writer hasn't heard of agents.

jacquesm 6mo ago

This article would be a lot more digestible if we didn't have actual scary data rather than just stories. Not a day goes by without some prompt injection oopsie, security gotcha, deepfake or some sandbox escape artist demonstration and tbh I'm impressed but more to the point where I don't doubt this is dangerous tech, I'm sure of it. This is roughly 1995 again and we're going to find out all over why mixing instructions and data was a spectacularly bad idea. Only now with human language as the input stream, which is far more expressive than HTML or SQL ever were. So now everybody is a hacker. At least in that sense it has leveled the playing field I guess.

FatherOfCurses 6mo ago

We tell ourselves scary stories about AI because humanity is rife with stories where a new idea or technology has had unintended negative consequences. AI bros just care about selling their product to another company and cashing out, they have absolutely no regard for their legacy.

cbogie 6mo ago
we tell ourselves scary stories because that’s what we do with language. and now our language isn’t just ours anymore.

the_af 6mo ago

I think the most insightful bit is buried in the article: > Perhaps because this is the best advertising money can’t buy. People like Harari and others repeat these accounts like ghost stories around a campfire. The public, awed and afraid, marvels at the capabilities of AI. And that's mostly it. PR. Publicity. Fear is good publicity if it emphasizes AI's capabilities. And people like Harari (or Gladwell) tell interesting and awe-inspiring stories that do not necessarily have much rigor or fact-checking in them. They simplify for storytelling purposes, which can result in misleading stories. I am worried about AI, but not about superintelligent AI that will exterminate or enslave us. I'm worried about AI as a tool to concentrate wealth and power in the hands of the current amoral entrepreneurial elite. I'm not sure whether I trust ChatGPT, but I sure as hell do NOT trust Sam Altman et al. Or, in other words, I subscribe to Ted Chiang's very apt remark about what we really fear: > “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker, “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.”

Swoerd 6mo ago

[dead]

netdevphoenix 6mo ago

There is a very interesting book that explores the West's generally negative view of artificial intelligence whenever it is portrayed in media (Skynet) while Japanese media tends to have a positive view (Astro Boy).

yanis_t 6mo ago

I wish we didn't call this AI as the term is crazily overloaded. Those are programs. The only difference is how we write them. Not with "if"s and "for"s. We take a bunch of bits that do nothing. Then we organize them in a way so that it outputs whatever it is we want.

mitthrowaway2 6mo ago
What kind of AI do you envision that would ever not fit that description?

RivieraKid 6mo ago

I 100% agree with this take, I find AI completely non-scary, especially in the sense that it's some kind of a conscious entity that will want to take over. I find these people almost delusional. It's a powerful tool, so it can be dangerous if used by people with bad intentions, so there's some real danger here, but my intuition is that it will be fine. The ratio between the power of people with good vs bad intentions shouldn't change too dramatically. The only scary part is that it could be bad for my future as a software developer. That said, I think it will be net benefit for the average worker - the average person will work less and earn more.

dclowd9901 6mo ago

My favorite part of this article was this bit, and naturally so, since I love the author: > Where did we come up with this caricature of AI’s obsessive rationality? “There’s an article I love by [the sci-fi author] Ted Chiang,” Mitchell said, “where he asks: What entity adheres monomaniacally to one single goal that they will pursue at all costs even if doing so uses up all the resources of the world? A big corporation. Their single goal is to increase value for shareholders, and in pursuing that, they can destroy the world. That’s what people are modeling their AI fantasies on.” As Chiang put it in the article in The New Yorker(opens a new tab), “Capitalism is the machine that will do whatever it takes to prevent us from turning it off.” I didn't realize it til I read it here, but yes, my fear isn't really about the machine, it's about the machine that drives the machine. We already have a class of amoral beings that treat the world as an expendable thing and are willing to burn it down for profit. We should focus on getting rid of that problem first.

greton7 6mo ago

[dead]

latentsea 6mo ago

Humans are the existence proof we seem to be chasing and the goal appears to be to hit a superset of humanities collective capabilities. I don't know if you've seen what we do to each other but... I'm afraid of a decent chunk of us too. If what we build is a superset of our capabilities then I am afraid of it, because I am already afraid of us.

garte 6mo ago

Great article, most of all the latter parts about what autopoiesis means regarding computers.

benj111 6mo ago

People were worried about trains in tunnels 180 years ago. People worry about new things. See also: the internet, video games, mobile phones, various genres of music. http://www.boxpeopleandplaces.co.uk/building-box-tunnel.html http://www.boxpeopleandplaces.co.uk/building-box-tunnel.html

ladax72707 6mo ago

> It occurred to me that if I could invent a machine - a gun - which could by its rapidity of fire, enable one man to do as much battle duty as a hundred, that it would, to a large extent supersede the necessity of large armies, and consequently, exposure to battle and disease [would] be greatly diminished.

xemdetia 6mo ago

The only thing I find scary about the current AI is how many AI companies became completely untethered from basic ethical controls openly. There is no facade of decency and it seems like a lot of them are running from their own shadow. I feel like there is a middle ground that could have been taken to bring content experts in instead of looting the web the world has built.

tim333 6mo ago

>After talking to experts, I was convinced there’s no reason to fear AIs developing a will to live, and then tricking or destroying us to avoid shutdown and take over the world. Unless, of course, we tell them to. Once we have super intelligent AIs I give it a day or two till someone tries that, prompting something like take over the world for me.

hateful 6mo ago

This may be pure conjecture, but my first thought is always that it might be a marketing ploy by the AI companies to make their products seem more advanced than they really are.

qgin 6mo ago

We’ve seen an incredibly powerful technology follow multiple exponential curves in its capability, but we’re supposed to ask why we’re telling ourselves “stories” if we think about what will happen if that technology continues to follow the curves it has been following without sign of hitting any walls? Is AGI certain? No. But there’s currently no specific reason to believe it isn’t coming in the next few years.