4 ms·

That's more of a blog article than a research paper... Scott Alexander (one of the writers), yudkowsky, and the others (not the other authors, the other group

That's more of a blog article than a research paper... Scott Alexander (one of the writers), yudkowsky, and the others (not the other authors, the other group of "thinkers" with similar ideas) are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Especially when that blog page is formatted in a deceptive way to look like a research paper It's not science, it's science fiction

ben_w 2y ago
> are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Weather vs. climate. The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). This is why a lot (I don't want to say "majority") of people who do have an actual background in machine learning and AI, pay attention to doomer arguments. Some of them* may be business leaders using the same language to BS their way into regulatory capture, but my experience of "real" AI researchers is they're mostly also "safety is important, Yudkowsky makes good points about XYZ" even if they would also say "my P(doom) is only 10%, not 95% like Yudkowsky". * I'm mainly thinking of Musk here, thanks to him saying "AI is summoning the demon" while also having an AI car company, funding OpenAI in the early years and now being in a legal spat with it that looks like it's "hostile takeover or interfere to the same end", funding another AI company, building humanoid robots and showing off ridiculous compute hardware, having brain implant chips, etc.
- amarcheschi 2y ago
  >The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. I'm relieved that at least in italy I still have to find someone in Ai taking them into consideration for more than a few minutes during an ethics course (with students sneering at the ideas of bostrom possible futures), and again, it's held by a professor with no technical knowledge with whom i often disagree due to this
  - ben_w 2y ago
    > But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. The base knowledge is game theory, not quite the same focus as the maths used to build an AI. And the problem isn't limited to "build god" — hence my examples of cobra effect, in which humans bred snakes because they were following the natural incentives of laws made by other humans who didn't see what would happen until it was so late that even cancelling the laws resulted in more snakes than they started with. > And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. And every "be careful what you wish for" story. Is climate change a potentially existential threat? Is global thermonuclear war a potentially existential threat? Are pandemics, both those from lab leaks and those evolving naturally in wet markets, potentially existential threats? The answer to all is "yes", even though these are systems with humans in the loop. (Even wet markets: people have been calling for better controls of them since well before Covid). AI is automation. Automation has bugs. If the automation has a lot of bugs, you've got humans constantly checking things, despite which errors still gets past QA from time to time. If it's perfect automation, you wouldn't have to check it… but nobody knows how to do perfect automation. "Perfect" automation would be god-like, but just as humans keep mistaking natural phenomena for deities, an AI doesn't have to actually be perfect for humans to set it running without checking the output and then be surprised when it all goes wrong. A decade ago the mistakes were companies doing blind dictionary merges on "Keep Calm and …" T-shirts, today it's LLMs giving legal advice (and perhaps writing US trade plans). They (the humans) shouldn't be doing those things, but they do them anyway, because humans are like that.
    - amarcheschi 2y ago
      My issue is not related to studying ai risk, my issue is empowering people who don't have formal education in anything related to ai. And yes, you need some math background otherwise you end up like yudkowski saying 3 years ago we all might be dead by now or next year. Or the use of bayesian probability in such a way thay makes you think they should have used their time better and followed a statistics course. There are ai researchers, serious ones, studying ai risk, and i don't see anything wrong in that. But of course, their claims and papers are way less, less alarmistic than the ai doomerism present in those circles. And one thing they sound the alarm on is the doomerism and the tescreal movement and ideals proposed by the aforementioned alexander, yud, bostrom ecc
- fc417fc802 2y ago
  > Some of them* may be business leaders using the same language to BS their way into regulatory capture Realistically, probably yeah. On the other hand, if you manage to occupy the high ground then you might be able to protect yourself. P( doom ) seems quite murky to me because conquering the real world involves physical hardware. We've had billions of general intelligences crawling all over the world waging war with one another for a while now. I doubt every single AGI magically ends up aligned in a common bloc against humanity; all the alternatives to that are hopelessly opaque. The worst case scenario that seems reasonably likely to me is probably AGI collectively not caring about us and wanting some natural resources that we happen to be living on top of.
  - amarcheschi 2y ago
    What if we create agi but then it hates existing and punish everyone who made it possible to exist? And i could go on for hours inventing possible reasons for similar roko basilisk the inverted basilisk. You created an agi. but that's the wrong one. it's literally the devil. game over you invented agi, but it likes pizza and its going to consume the entire universe to make pizza. game over, but at least you'll eat pizza til the end you invented agi, but its depressed and refuses to actually do something. you spent huge amount of resources and all you have is a chatbot that tells you to leave it alone. you don't invent agi, it's not possible. i'm hearing from here the VCs cry you invented agi, but it decides the only language it wants to use is a language it invented, and you have no way to understand how to interact with it. Great, agi is a non verbal autistic agi. And well, one could continue for hours in the most hilarious way that not necessarily go in the direction of doom, but of course the idea of doom is going to have a wider reach. then you read yudkwosky thoughts about how it would kill everyone with nanobots and you realize you're reading a science fiction piece. a bad one. at least neuromancer was interesting
    - fc417fc802 2y ago
      My prediction is approximately that all of the above get created in various quantity by different groups of people at approximately the same time. Hardware fabrication in the real world is a (relatively) slow process. In such a scenario it seems far from certain that a single AGI would decisively gain the upper hand. The ensuing chaos seems completely impossible to predict much of anything about.
frozenseven 2y ago
If you're after those oh-so precious credentials, Bengio, Hinton and Russell have been echoing much of the same statements. Yoshua Bengio even shared the "AI 2027" article, recommending that people read it. Also, pretty laughable you cite Timnit Gebru as any type of "researcher". She's a political activist with precisely zero contributions to hard science.

ben_w 2y ago

> are more or less Ai doomers with no actual background in machine learning/ai I don't think why we should listen to them. Weather vs. climate. The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). This is why a lot (I don't want to say "majority") of people who do have an actual background in machine learning and AI, pay attention to doomer arguments. Some of them* may be business leaders using the same language to BS their way into regulatory capture, but my experience of "real" AI researchers is they're mostly also "safety is important, Yudkowsky makes good points about XYZ" even if they would also say "my P(doom) is only 10%, not 95% like Yudkowsky". * I'm mainly thinking of Musk here, thanks to him saying "AI is summoning the demon" while also having an AI car company, funding OpenAI in the early years and now being in a legal spat with it that looks like it's "hostile takeover or interfere to the same end", funding another AI company, building humanoid robots and showing off ridiculous compute hardware, having brain implant chips, etc.

amarcheschi 2y ago
>The question they're asking isn't about machine learning specifically, it's about the risks of generic optimisers optimising a utility function, and the difficulty of specifying a utility function in a way that doesn't have unfortunate side effects. The examples they give also work with biology (genetics and the difference between what your genes "want" and what your brain "wants") and with governance (laws and loopholes, cobra effects, etc.). But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. I'm relieved that at least in italy I still have to find someone in Ai taking them into consideration for more than a few minutes during an ethics course (with students sneering at the ideas of bostrom possible futures), and again, it's held by a professor with no technical knowledge with whom i often disagree due to this
- ben_w 2y ago
  > But you do need some kind of base knowledge, if you want to talk about this. Otherwise you're saying "what if we create God". And last time I checked it wasn't possible. The base knowledge is game theory, not quite the same focus as the maths used to build an AI. And the problem isn't limited to "build god" — hence my examples of cobra effect, in which humans bred snakes because they were following the natural incentives of laws made by other humans who didn't see what would happen until it was so late that even cancelling the laws resulted in more snakes than they started with. > And what's with the existential risk obsession? That's like a bad retelling of the Pascal bet on the existence of God. And every "be careful what you wish for" story. Is climate change a potentially existential threat? Is global thermonuclear war a potentially existential threat? Are pandemics, both those from lab leaks and those evolving naturally in wet markets, potentially existential threats? The answer to all is "yes", even though these are systems with humans in the loop. (Even wet markets: people have been calling for better controls of them since well before Covid). AI is automation. Automation has bugs. If the automation has a lot of bugs, you've got humans constantly checking things, despite which errors still gets past QA from time to time. If it's perfect automation, you wouldn't have to check it… but nobody knows how to do perfect automation. "Perfect" automation would be god-like, but just as humans keep mistaking natural phenomena for deities, an AI doesn't have to actually be perfect for humans to set it running without checking the output and then be surprised when it all goes wrong. A decade ago the mistakes were companies doing blind dictionary merges on "Keep Calm and …" T-shirts, today it's LLMs giving legal advice (and perhaps writing US trade plans). They (the humans) shouldn't be doing those things, but they do them anyway, because humans are like that.
  - amarcheschi 2y ago
    My issue is not related to studying ai risk, my issue is empowering people who don't have formal education in anything related to ai. And yes, you need some math background otherwise you end up like yudkowski saying 3 years ago we all might be dead by now or next year. Or the use of bayesian probability in such a way thay makes you think they should have used their time better and followed a statistics course. There are ai researchers, serious ones, studying ai risk, and i don't see anything wrong in that. But of course, their claims and papers are way less, less alarmistic than the ai doomerism present in those circles. And one thing they sound the alarm on is the doomerism and the tescreal movement and ideals proposed by the aforementioned alexander, yud, bostrom ecc
fc417fc802 2y ago
> Some of them* may be business leaders using the same language to BS their way into regulatory capture Realistically, probably yeah. On the other hand, if you manage to occupy the high ground then you might be able to protect yourself. P( doom ) seems quite murky to me because conquering the real world involves physical hardware. We've had billions of general intelligences crawling all over the world waging war with one another for a while now. I doubt every single AGI magically ends up aligned in a common bloc against humanity; all the alternatives to that are hopelessly opaque. The worst case scenario that seems reasonably likely to me is probably AGI collectively not caring about us and wanting some natural resources that we happen to be living on top of.
- amarcheschi 2y ago
  What if we create agi but then it hates existing and punish everyone who made it possible to exist? And i could go on for hours inventing possible reasons for similar roko basilisk the inverted basilisk. You created an agi. but that's the wrong one. it's literally the devil. game over you invented agi, but it likes pizza and its going to consume the entire universe to make pizza. game over, but at least you'll eat pizza til the end you invented agi, but its depressed and refuses to actually do something. you spent huge amount of resources and all you have is a chatbot that tells you to leave it alone. you don't invent agi, it's not possible. i'm hearing from here the VCs cry you invented agi, but it decides the only language it wants to use is a language it invented, and you have no way to understand how to interact with it. Great, agi is a non verbal autistic agi. And well, one could continue for hours in the most hilarious way that not necessarily go in the direction of doom, but of course the idea of doom is going to have a wider reach. then you read yudkwosky thoughts about how it would kill everyone with nanobots and you realize you're reading a science fiction piece. a bad one. at least neuromancer was interesting
  - fc417fc802 2y ago
    My prediction is approximately that all of the above get created in various quantity by different groups of people at approximately the same time. Hardware fabrication in the real world is a (relatively) slow process. In such a scenario it seems far from certain that a single AGI would decisively gain the upper hand. The ensuing chaos seems completely impossible to predict much of anything about.

frozenseven 2y ago

If you're after those oh-so precious credentials, Bengio, Hinton and Russell have been echoing much of the same statements. Yoshua Bengio even shared the "AI 2027" article, recommending that people read it. Also, pretty laughable you cite Timnit Gebru as any type of "researcher". She's a political activist with precisely zero contributions to hard science.