4 ms·

Yudkowsky is a philosopher, in the sense of someone who thinks a lot about things that haven't been achieved yet. Lots of otherwise smart people (wrongly!) disc

by marvin 5y ago

Yudkowsky is a philosopher, in the sense of someone who thinks a lot about things that haven't been achieved yet. Lots of otherwise smart people (wrongly!) discount the value of philosophy, but it's close by every time there's a paradigm shift in humanity's knowledge. Philosophers can be scientists and vice versa. If anything, I'm surprised that this philosophy isn't mentioned more in a thread where the author gleefully talks about ML being used to create better AI, layer by layer until the thing is even more opaque than what we're currently working with. This is terrifying, as we currently have only very loose ideas about how to reliably ensure that a powerful reinforcement learning system doesn't accidentally optimize for something we don't want. The current paradigm is "turn it off", which works well for now but seems like a fragile shield long-term.

the_af 5y ago
> Yudkowsky is a philosopher At least inasmuch as anyone who thinks about stuff can be considered a philosopher. But he strikes me much more as a self-appointed expert on matters where he shows no achievements. He writes fanfiction about AI rather than actually doing stuff with AI.
- psyc 5y ago
  Most people who think about stuff don’t leave a mountain of highly organized and entertaining essays for posterity.
  - the_af 5y ago
    That just means he's a prolific writer, which I never questioned. I'm arguing that what he writes is fanfiction (which he takes way too seriously), and that he's not an expert in AI and therefore we shouldn't take his predictions too seriously.
    - franknstein 5y ago
      There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future. Sometimes You don't have to be an expert to notice the trends in certain domains. I'd go even further - it could be easier to notice the big picture without being overly bothered with nitty-gritty technical details - yes, to further the field they are necessary, but to comment the direction the field is heading in and about its implications for society they are not. I don't necessarily agree with the author, I'm just making a general comment.
      - the_af 5y ago
        > "There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future" That's a pretty weak argument. I love scifi, and I love for example Philip Dick's writing, yet I would not consider PKD's opinion on the future of AI/AGI particularly relevant. James Cameron is not an authority on AI either. If people said "Yudkowsky is a nice fanfiction author" it would be one thing. But he considers himself an actual AI researcher, and that's just not right. He is not qualified, and has no accomplishments in the area, other than writing fanfiction about it.
        psyc 5y ago
        You keep hammering on with the same lazy slander. Yudkowsky was well known long before he wrote the popular Harry Potter fanfic, which incidentally is pedagogical / allegorical. Because his main roles are teacher and philosopher, and philosophers do that. Here's a sampling of his non-fiction writing: https://www.lesswrong.com/tag/sequences https://www.lesswrong.com/tag/sequences https://intelligence.org/files/EthicsofAI.pdf https://intelligence.org/files/EthicsofAI.pdf https://arxiv.org/abs/1710.05060 https://arxiv.org/abs/1710.05060 https://intelligence.org/files/AlignmentMachineLearning.pdf https://intelligence.org/files/AlignmentMachineLearning.pdf https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577 https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/DefinabilityTruthDraft.pdf https://intelligence.org/files/DefinabilityTruthDraft.pdf https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf https://intelligence.org/files/TilingAgentsDraft.pdf https://intelligence.org/files/TilingAgentsDraft.pdf But you'll just say there's nothing of value there, and it's somehow figuratively "fan fiction", because he didn't go to college, and he doesn't work much on ML, which is clearly the end-all of AI.
goatlover 5y ago
Nick Bostrom is a professional philosopher who has also thought a lot about AGI and simulation scenarios. But just because educated, smart people can think a lot about something doesn't it mean it will necessarily happen. I'm sure there's quite a few people who have thought at length about warp drives and wormholes. That doesn't mean we'll ever be able to make use of them.
- gizajob 5y ago
  Nick Bostrom is also an egotistical grinder, who couldn't programme a microwave.
  - TheRealNGenius 5y ago
    I agree, that guy's also a whole lotta bad takes put all together. Like a slutty potato.
ASalazarMX 5y ago
In the few contact I've had with philosophers, I've got the (maybe wrong and oversimplified, but nonetheless evident) impression that they find philosophy fun and mentally stimulating, and they enjoy academically talking and writing about edge cases no matter their feasibility. I think it's a net good when philosophers mentally explore scenarios most people don't consider, regardless of their plausibility, but they will not be taken more seriously that Peter Watts looking for inspiration for his next scifi book unless they offer some kind of evidence to sustain their conclusions. The handful of philosophers I've known are very smart, interesting people, but except for one who teaches philosophy, they don't live off philosophy. Most have a side job, and one lives with his parents. It's as if philosophy was more of a mentally rewarding hobby than a job.
- tuatoru 5y ago
  Mentally rewarding hobbies can be fruitful, though. Consider the case of Oliver Heaviside, the person who came up with Maxwell's equations of electromagnetism in the form that we use today, as well as several other useful things. I have no doubt that he was thought a crank by many, nor that many other actual cranks were claiming discoveries based on Maxwell's work. One can only tell the difference in the rear-view mirror. 1. https://en.wikipedia.org/wiki/Oliver_Heaviside https://en.wikipedia.org/wiki/Oliver_Heaviside
  - marvin 5y ago
    Why do people have to use the derogatory word «crank»? It’s perfectly possible to just be smart, sincere and wrong. Talking about paradigm shifts again, not a single one of them matched the consensus at the time. Isaac Newton spent a lot of time thinking about alchemy and religion, for heaven’s sake. During his lifetime, it wasn’t obvious even to the smartest thinkers whether science, the Bible or quasi-religious rituals was the best tool for understanding the world. Conventional thinkers expect that the future will look like what they know, and it leads to frequent, overconfident dismissal of everything that is unconventional. By all means, disagree and explore in your own direction. But don’t go around degrading those you disagree with. It’s just so average.

by marvin 5y ago

the_af 5y ago

> Yudkowsky is a philosopher At least inasmuch as anyone who thinks about stuff can be considered a philosopher. But he strikes me much more as a self-appointed expert on matters where he shows no achievements. He writes fanfiction about AI rather than actually doing stuff with AI.

psyc 5y ago
Most people who think about stuff don’t leave a mountain of highly organized and entertaining essays for posterity.
- the_af 5y ago
  That just means he's a prolific writer, which I never questioned. I'm arguing that what he writes is fanfiction (which he takes way too seriously), and that he's not an expert in AI and therefore we shouldn't take his predictions too seriously.
  - franknstein 5y ago
    There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future. Sometimes You don't have to be an expert to notice the trends in certain domains. I'd go even further - it could be easier to notice the big picture without being overly bothered with nitty-gritty technical details - yes, to further the field they are necessary, but to comment the direction the field is heading in and about its implications for society they are not. I don't necessarily agree with the author, I'm just making a general comment.
    - the_af 5y ago
      > "There is long history of science-fiction writers painting correct visions of, at the time seemingly impossible future" That's a pretty weak argument. I love scifi, and I love for example Philip Dick's writing, yet I would not consider PKD's opinion on the future of AI/AGI particularly relevant. James Cameron is not an authority on AI either. If people said "Yudkowsky is a nice fanfiction author" it would be one thing. But he considers himself an actual AI researcher, and that's just not right. He is not qualified, and has no accomplishments in the area, other than writing fanfiction about it.
      - psyc 5y ago
        You keep hammering on with the same lazy slander. Yudkowsky was well known long before he wrote the popular Harry Potter fanfic, which incidentally is pedagogical / allegorical. Because his main roles are teacher and philosopher, and philosophers do that. Here's a sampling of his non-fiction writing: https://www.lesswrong.com/tag/sequences https://www.lesswrong.com/tag/sequences https://intelligence.org/files/EthicsofAI.pdf https://intelligence.org/files/EthicsofAI.pdf https://arxiv.org/abs/1710.05060 https://arxiv.org/abs/1710.05060 https://intelligence.org/files/AlignmentMachineLearning.pdf https://intelligence.org/files/AlignmentMachineLearning.pdf https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577 https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/DefinabilityTruthDraft.pdf https://intelligence.org/files/DefinabilityTruthDraft.pdf https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf https://intelligence.org/files/TilingAgentsDraft.pdf https://intelligence.org/files/TilingAgentsDraft.pdf But you'll just say there's nothing of value there, and it's somehow figuratively "fan fiction", because he didn't go to college, and he doesn't work much on ML, which is clearly the end-all of AI.

goatlover 5y ago

Nick Bostrom is a professional philosopher who has also thought a lot about AGI and simulation scenarios. But just because educated, smart people can think a lot about something doesn't it mean it will necessarily happen. I'm sure there's quite a few people who have thought at length about warp drives and wormholes. That doesn't mean we'll ever be able to make use of them.

gizajob 5y ago
Nick Bostrom is also an egotistical grinder, who couldn't programme a microwave.
- TheRealNGenius 5y ago
  I agree, that guy's also a whole lotta bad takes put all together. Like a slutty potato.

ASalazarMX 5y ago

In the few contact I've had with philosophers, I've got the (maybe wrong and oversimplified, but nonetheless evident) impression that they find philosophy fun and mentally stimulating, and they enjoy academically talking and writing about edge cases no matter their feasibility. I think it's a net good when philosophers mentally explore scenarios most people don't consider, regardless of their plausibility, but they will not be taken more seriously that Peter Watts looking for inspiration for his next scifi book unless they offer some kind of evidence to sustain their conclusions. The handful of philosophers I've known are very smart, interesting people, but except for one who teaches philosophy, they don't live off philosophy. Most have a side job, and one lives with his parents. It's as if philosophy was more of a mentally rewarding hobby than a job.

tuatoru 5y ago
Mentally rewarding hobbies can be fruitful, though. Consider the case of Oliver Heaviside, the person who came up with Maxwell's equations of electromagnetism in the form that we use today, as well as several other useful things. I have no doubt that he was thought a crank by many, nor that many other actual cranks were claiming discoveries based on Maxwell's work. One can only tell the difference in the rear-view mirror. 1. https://en.wikipedia.org/wiki/Oliver_Heaviside https://en.wikipedia.org/wiki/Oliver_Heaviside
- marvin 5y ago
  Why do people have to use the derogatory word «crank»? It’s perfectly possible to just be smart, sincere and wrong. Talking about paradigm shifts again, not a single one of them matched the consensus at the time. Isaac Newton spent a lot of time thinking about alchemy and religion, for heaven’s sake. During his lifetime, it wasn’t obvious even to the smartest thinkers whether science, the Bible or quasi-religious rituals was the best tool for understanding the world. Conventional thinkers expect that the future will look like what they know, and it leads to frequent, overconfident dismissal of everything that is unconventional. By all means, disagree and explore in your own direction. But don’t go around degrading those you disagree with. It’s just so average.