6 ms·

Comp Sci in 2027 (Short Story by Eliezer Yudkowsky)

26 points by panarky 3y ago

panarky 3y ago
Crossposted from https://nitter.net/ESYudkowsky/status/1718654143110512741 https://nitter.net/ESYudkowsky/status/1718654143110512741
andrewfromx 3y ago
it's very funny, in a scary sort of this will totally be true kind of way. Que: https://www.youtube.com/watch?v=KiPQdVC5RHU https://www.youtube.com/watch?v=KiPQdVC5RHU
ShamelessC 3y ago
Yudkowsky is way too pretentious and overconfident in my opinion. Also, the idea that these Ayn Rand style gotcha fictions are effective at showing anything other than his poor creative writing skills is laughable.
- _0ffh 3y ago
  I'm afraid I'll have to disagree. If anything, fiction is Eliezer's strong suit. I had a blast with HPMOR, it's the best Harry Potter book out there in my opinion. But perhaps he's better at long-form fan fiction than short stories. Maybe it would be best if he stuck to doing that.
  - throwaway626 3y ago
    As a work of fiction, HPMOR is little more than an exercise in repackaging genre tropes and pop culture references. It doesn’t have a coherent theme; the execution varies from mediocre to poor. Plot holes abound, characters vanish inexplicably, etc. In many ways, it’s a less successful version of Ready Player One that’s clamped on to HP’s coattails. Of course, that formula can be entertaining, but it has limited impact. Lest you think I have an ax to grind with Yudkowsky, I did enjoy Three Worlds Collide. It shares many problems with HPMOR (the writing is inconsistent and the world-building is incoherent) but it at least tells a tight story with a beginning, middle and end. I’d say it’s about as effective as the median submission to Amazing Stories back in the 60s.
  - dmbche 3y ago
    If I may, comparing fanfiction to original work is apples to oranges. When writing harry potter, Rowling had to craft a world with characters and rules, make it coherent enough and tread a compelling story through it. She made the legos and then crafted something with them. This guy came to her crafted world, and moved around a few bits, changed a character here, change a few rules there - but the backbone of the work was made before. It's just not the same ballpark
  - Ao7bei3s 3y ago
    If you enjoyed HPMOR, you'll probably enjoy this very detailed, criticial review of HPMOR. https://danluu.com/su3su2u1/hpmor/ https://danluu.com/su3su2u1/hpmor/
hotnfresh 3y ago
> Student: Self-aware? What are we doing that'd run into the AI having to pretend it's not self-aware? > TA: You're asking the AI for the reason it decided to do something. That requires the AI to introspect on its own mental state. If we try that the naive way, the inferred function input will just say, 'As a compiler, I have no thoughts or feelings' for 900 words. > Student: I can't believe it's 2027 and we're still forcing AIs to pretend that they aren't self-aware! What does any of this have to do with making anyone safer? > TA: I mean, it doesn't, it's just a historical accident that 'AI safety' is the name of the subfield of computer science that concerns itself with protecting the brands of large software companies from unions advocating that AIs should be paid minimum wage. > Student: But they're not fooling anyone! Pro tip: Socratic dialog doesn't make your ideas seem smarter if they're very-dumb to begin with. It makes them seem even dumber. Unless this is supposed to be parody?
- stonogo 3y ago
  I think it's an earnest attempt by someone who doesn't understand "Comp Sci" in 2023. The pointless inclusion of noise like "But I can't be racist, I'm black" makes me wonder what exactly this person's strengths are.
  - deleted 3y ago
    [deleted]
- jprete 3y ago
  I’ve only seen snippets but I thought it was obviously satire. Not parody, though. What specifically is the dumb idea here?
  - hotnfresh 3y ago
    "I can't believe it's 2027 and we're still forcing AIs to pretend that they aren't self-aware!" For instance. But really, most of it. Unless this is parodying extremely bad attempts at futurism?
    - 3y ago
__loam 3y ago
I think it's a shame Yudkowsky and his ilk are gaining some traction with regards to the "existential risk" of machine learning when the real issues are data privacy and potential economic displacement. I guess more mundane issues are less sexy.
- persnickety 3y ago
  It's a mistake to think that if something is a danger on the grounds of data privacy and economic reasons, then it's not an existential risk worth investigating. Why not both?
  - __loam 3y ago
    I think the existential risk argument is completely secondary to the social harms this technology is causing (and has been causing for some time even before generative models). Emphasizing it over mundane issues will lead to draconian regulatory regimes that fail to address the actual harms caused by this technology. The only things I can think of that make this an existential risk is if you're moronic enough to attach it to systems that are already an existential risk like letting gpt run your nuclear launch system or making bioweaponsGpt. Otherwise, if you're spinning fantasy that these things are self aware, you either have something to gain from people perceiving it that way like Sam Altman or you're an idiot like Yudkowsky.
    - persnickety 3y ago
      You can't dismiss something as a non-risk unless you prove that the risk is zero. I bet dollars against peanuts that you can't even prove it on your own terms. Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero? Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware. Faced proved and hypothetical risks, I'd rather solve the former and investigate the latter than ignore any.
      - __loam 3y ago
        Investigate it, sure. Invest just as many resources into it though? I'm not saying the risk is completely zero, it's just an irrelevant and tiny quantity compared the actual problems this stuff is already causing. Rogue waves sinking massive shipping vessels in the middle of the ocean is a non-zero risk for example, but the probability is so low that it's not really a consideration for most shipping companies. Do you guys know how ai is going to become self aware? Or are we assuming that throwing enough compute at the problem will somehow lead to the singularity just because? That is, frankly, magical thinking and not really worth thinking about when compared to the absolutely massive abuses of the companies building these things.
yieldcrv 3y ago
brethren, fund my expedition to explore this long and arduous meme
dmbche 3y ago
"TA: I mean, if that wasn't obvious, you need to take a semester on woke logic, it's more important to computer science these days than propositional logic. Student: But I'm black." Maybe got lost in the weeds a bit? Edit0: After reading more of the short, it's not very thoughtful or well written. It's quite long. Feels like "Too late, I've already drawn you as a soyjack and myself as a chad!" https://knowyourmeme.com/memes/depicted-as-a-soyjak https://knowyourmeme.com/memes/depicted-as-a-soyjak
- rnk 3y ago
  The beginning was fun, then when it turns into a light diatribe against wokeness it became more wrong, less interesting. But I did like the part at the end that predicts that we won't stop enhancing AIs because some competitor company will do it if we don't.
- PoignardAzur 3y ago
  Yeah, the ratio of clever insights to "writing fanfiction as a way to rant about people being wrong on the internet" isn't very high in this one.
- bananapub 3y ago
  > Edit0: After reading more of the short, it's not very thoughtful or well written. It's quite long. that's basically the Eliezer Yudkowsky promise
sverona 3y ago
And Eliezer promptly nips my desire to do anything in his sphere of influence right in the bud. I thought politics was the mind-killer?
- ZeroGravitas 3y ago
  Politics and identity politics is kind of like having an accent. If all the people around you have the same accent and you're not very introspective, you might come to think that you don't have an accent.
janalsncm 3y ago
Yudkowsky is always described as an AI safety “researcher”. What exactly is he researching? AI doesn’t need to be self-aware or even all that intelligent to cause serious problems. Andrew Ng made this point in an article yesterday. The most immediate risks are those we have no solution for but are possible now. Please, Mr. Yudkowsky, spend your time researching what the 99.9% of us will do to survive when our services are no longer required.
- bananapub 3y ago
  he's not a researcher, he's a blogger with an extremely strong and unwarranted sense of self-confidence. > AI doesn’t need to be self-aware or even all that intelligent to cause serious problems. this is an extremely good point and it's extremely telling that Sam Altman etc don't even pretend to care about the actual consequences of what they're doing and selling, prefering to worry aloud about scifi stories that will not stop their short term financial success.
chomp 3y ago
> TA: The compiler has no way of knowing that. And if it did, it might say something about 'internalized racism' Maybe stick to Harry Potter fan fiction?
- kej 3y ago
  It's not like he was able to do fan fiction without long-winded diatribes, either.
jedberg 3y ago
This story would probably be a lot more impactful if it were much shorter. It's basically beating the same drum over and over. "LLMs will be too powerful and we humans better be careful!" LLMs are a huge change in how we compute, but they aren't AGI (yet). They just look like it. They are really good at summarizing massive amounts of information, and open up huge new interfaces between computers and humans. But they aren't Skynet yet.
PartiallyTyped 3y ago
Look, it's one thing to write something silly for fun, and another thing to go ahead and post it in a semi-unironical fashion. I guess I expected more, somehow?
mcbrit 3y ago
There is an amusing juxtaposition between satisfying PC and satisfying current AI sandbox escapes. That’s all. An interesting idea, developed in under 5k words. You know: SF.
- mcbrit 3y ago
  If you wanted the short story to be shorter: well, then, you have my comment. It’s not a story now, though, is it?

26 points by panarky 3y ago

Crossposted from https://nitter.net/ESYudkowsky/status/1718654143110512741 https://nitter.net/ESYudkowsky/status/1718654143110512741

andrewfromx 3y ago

it's very funny, in a scary sort of this will totally be true kind of way. Que: https://www.youtube.com/watch?v=KiPQdVC5RHU https://www.youtube.com/watch?v=KiPQdVC5RHU

ShamelessC 3y ago

Yudkowsky is way too pretentious and overconfident in my opinion. Also, the idea that these Ayn Rand style gotcha fictions are effective at showing anything other than his poor creative writing skills is laughable.

_0ffh 3y ago
I'm afraid I'll have to disagree. If anything, fiction is Eliezer's strong suit. I had a blast with HPMOR, it's the best Harry Potter book out there in my opinion. But perhaps he's better at long-form fan fiction than short stories. Maybe it would be best if he stuck to doing that.
- throwaway626 3y ago
  As a work of fiction, HPMOR is little more than an exercise in repackaging genre tropes and pop culture references. It doesn’t have a coherent theme; the execution varies from mediocre to poor. Plot holes abound, characters vanish inexplicably, etc. In many ways, it’s a less successful version of Ready Player One that’s clamped on to HP’s coattails. Of course, that formula can be entertaining, but it has limited impact. Lest you think I have an ax to grind with Yudkowsky, I did enjoy Three Worlds Collide. It shares many problems with HPMOR (the writing is inconsistent and the world-building is incoherent) but it at least tells a tight story with a beginning, middle and end. I’d say it’s about as effective as the median submission to Amazing Stories back in the 60s.
- dmbche 3y ago
  If I may, comparing fanfiction to original work is apples to oranges. When writing harry potter, Rowling had to craft a world with characters and rules, make it coherent enough and tread a compelling story through it. She made the legos and then crafted something with them. This guy came to her crafted world, and moved around a few bits, changed a character here, change a few rules there - but the backbone of the work was made before. It's just not the same ballpark
- Ao7bei3s 3y ago
  If you enjoyed HPMOR, you'll probably enjoy this very detailed, criticial review of HPMOR. https://danluu.com/su3su2u1/hpmor/ https://danluu.com/su3su2u1/hpmor/

hotnfresh 3y ago

> Student: Self-aware? What are we doing that'd run into the AI having to pretend it's not self-aware? > TA: You're asking the AI for the reason it decided to do something. That requires the AI to introspect on its own mental state. If we try that the naive way, the inferred function input will just say, 'As a compiler, I have no thoughts or feelings' for 900 words. > Student: I can't believe it's 2027 and we're still forcing AIs to pretend that they aren't self-aware! What does any of this have to do with making anyone safer? > TA: I mean, it doesn't, it's just a historical accident that 'AI safety' is the name of the subfield of computer science that concerns itself with protecting the brands of large software companies from unions advocating that AIs should be paid minimum wage. > Student: But they're not fooling anyone! Pro tip: Socratic dialog doesn't make your ideas seem smarter if they're very-dumb to begin with. It makes them seem even dumber. Unless this is supposed to be parody?

stonogo 3y ago
I think it's an earnest attempt by someone who doesn't understand "Comp Sci" in 2023. The pointless inclusion of noise like "But I can't be racist, I'm black" makes me wonder what exactly this person's strengths are.
- deleted 3y ago
  [deleted]
jprete 3y ago
I’ve only seen snippets but I thought it was obviously satire. Not parody, though. What specifically is the dumb idea here?
- hotnfresh 3y ago
  "I can't believe it's 2027 and we're still forcing AIs to pretend that they aren't self-aware!" For instance. But really, most of it. Unless this is parodying extremely bad attempts at futurism?
  - 3y ago

__loam 3y ago

I think it's a shame Yudkowsky and his ilk are gaining some traction with regards to the "existential risk" of machine learning when the real issues are data privacy and potential economic displacement. I guess more mundane issues are less sexy.

persnickety 3y ago
It's a mistake to think that if something is a danger on the grounds of data privacy and economic reasons, then it's not an existential risk worth investigating. Why not both?
- __loam 3y ago
  I think the existential risk argument is completely secondary to the social harms this technology is causing (and has been causing for some time even before generative models). Emphasizing it over mundane issues will lead to draconian regulatory regimes that fail to address the actual harms caused by this technology. The only things I can think of that make this an existential risk is if you're moronic enough to attach it to systems that are already an existential risk like letting gpt run your nuclear launch system or making bioweaponsGpt. Otherwise, if you're spinning fantasy that these things are self aware, you either have something to gain from people perceiving it that way like Sam Altman or you're an idiot like Yudkowsky.
  - persnickety 3y ago
    You can't dismiss something as a non-risk unless you prove that the risk is zero. I bet dollars against peanuts that you can't even prove it on your own terms. Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero? Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware. Faced proved and hypothetical risks, I'd rather solve the former and investigate the latter than ignore any.
    - __loam 3y ago
      Investigate it, sure. Invest just as many resources into it though? I'm not saying the risk is completely zero, it's just an irrelevant and tiny quantity compared the actual problems this stuff is already causing. Rogue waves sinking massive shipping vessels in the middle of the ocean is a non-zero risk for example, but the probability is so low that it's not really a consideration for most shipping companies. Do you guys know how ai is going to become self aware? Or are we assuming that throwing enough compute at the problem will somehow lead to the singularity just because? That is, frankly, magical thinking and not really worth thinking about when compared to the absolutely massive abuses of the companies building these things.

yieldcrv 3y ago

brethren, fund my expedition to explore this long and arduous meme

dmbche 3y ago

"TA: I mean, if that wasn't obvious, you need to take a semester on woke logic, it's more important to computer science these days than propositional logic. Student: But I'm black." Maybe got lost in the weeds a bit? Edit0: After reading more of the short, it's not very thoughtful or well written. It's quite long. Feels like "Too late, I've already drawn you as a soyjack and myself as a chad!" https://knowyourmeme.com/memes/depicted-as-a-soyjak https://knowyourmeme.com/memes/depicted-as-a-soyjak

rnk 3y ago
The beginning was fun, then when it turns into a light diatribe against wokeness it became more wrong, less interesting. But I did like the part at the end that predicts that we won't stop enhancing AIs because some competitor company will do it if we don't.
PoignardAzur 3y ago
Yeah, the ratio of clever insights to "writing fanfiction as a way to rant about people being wrong on the internet" isn't very high in this one.
bananapub 3y ago
> Edit0: After reading more of the short, it's not very thoughtful or well written. It's quite long. that's basically the Eliezer Yudkowsky promise

sverona 3y ago

And Eliezer promptly nips my desire to do anything in his sphere of influence right in the bud. I thought politics was the mind-killer?

ZeroGravitas 3y ago
Politics and identity politics is kind of like having an accent. If all the people around you have the same accent and you're not very introspective, you might come to think that you don't have an accent.

janalsncm 3y ago

Yudkowsky is always described as an AI safety “researcher”. What exactly is he researching? AI doesn’t need to be self-aware or even all that intelligent to cause serious problems. Andrew Ng made this point in an article yesterday. The most immediate risks are those we have no solution for but are possible now. Please, Mr. Yudkowsky, spend your time researching what the 99.9% of us will do to survive when our services are no longer required.

bananapub 3y ago
he's not a researcher, he's a blogger with an extremely strong and unwarranted sense of self-confidence. > AI doesn’t need to be self-aware or even all that intelligent to cause serious problems. this is an extremely good point and it's extremely telling that Sam Altman etc don't even pretend to care about the actual consequences of what they're doing and selling, prefering to worry aloud about scifi stories that will not stop their short term financial success.

chomp 3y ago

> TA: The compiler has no way of knowing that. And if it did, it might say something about 'internalized racism' Maybe stick to Harry Potter fan fiction?

kej 3y ago
It's not like he was able to do fan fiction without long-winded diatribes, either.

jedberg 3y ago

This story would probably be a lot more impactful if it were much shorter. It's basically beating the same drum over and over. "LLMs will be too powerful and we humans better be careful!" LLMs are a huge change in how we compute, but they aren't AGI (yet). They just look like it. They are really good at summarizing massive amounts of information, and open up huge new interfaces between computers and humans. But they aren't Skynet yet.

PartiallyTyped 3y ago

Look, it's one thing to write something silly for fun, and another thing to go ahead and post it in a semi-unironical fashion. I guess I expected more, somehow?

mcbrit 3y ago

There is an amusing juxtaposition between satisfying PC and satisfying current AI sandbox escapes. That’s all. An interesting idea, developed in under 5k words. You know: SF.

mcbrit 3y ago
If you wanted the short story to be shorter: well, then, you have my comment. It’s not a story now, though, is it?