4 ms·

As usual, Yudkowsky has valuable insights, but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be.

by cs702 4y ago

As usual, Yudkowsky has valuable insights, but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be. TL;DR: If your customers can get text input-output samples from your super-secret LLM, whether via web interface or API, then your customers can easily and cheaply train their own LLM to learn to act like your super-secret one. For example, if you're OpenAI, your customers can spend ~$100 to get input-output samples from text-davinci-003, plus another ~$500 to fine-tune LLaMA on those samples, and -- voila! -- they get Alpaca.

PaulHoule 4y ago
It’s not accidental at all that EY writes this way. He is writing out of L. Ron Hubbard’s style guide. He doesn’t want to communicate to outsiders, he is writing to members of his Cthulhu cult.
- canadiantim 4y ago
  Yeah I was liking his post and following along until he kept repeating "shoggoth", very weird. I stopped at that part
  - deleted 4y ago
    [deleted]
  - 4bpp 4y ago
    The "shoggoth with a smiley face" metaphor for LLMs has gained traction far beyond Yudkowsky's bubble. It even has a KYM article now: https://knowyourmeme.com/memes/shoggoth-with-smiley-face-artificial-intelligence https://knowyourmeme.com/memes/shoggoth-with-smiley-face-art...
    - semi-extrinsic 4y ago
      Doesn't make any of this less stupid though.
      - chpatrick 4y ago
        Makes sense to me. It's an enormous inhuman blob trained on all of human knowledge that can manifest a personality out of nothing based on some inoffensive corporate prompt.
- jerry1979 4y ago
  What do you mean by "L. Ron Hubbard's style guide"? Would that be doing things like employing in-group metaphors?
  - PaulHoule 4y ago
    Making up lots of vocabulary that divides insiders from outsiders. I was interested in Scientology as a microcosm of society around the time alt.religion.scientology appeared in the early 1990s. Scientology documented everything about how it runs, even management is scriptural, and only Hubbard could write scripture so it is like a specimen that's been fixed to a microscope slide. Since then renegade "Free Zone" Scientologists have put all of his writings online so it is very accessible. Dianetics is the Sequences of Scientology in that it is a long, windy, and nonsensical book that selects for people who are perseverant readers who are impervious to critical thinking. (Like one of those Nigerian scam emails.) Hubbard's hypnotic language is demonstrated in his Philadelphia Doctorate Lectures where he not only mentions the hypnotist Milton Erickson but uses Erickson's techniques extensively. (There is no record of Hubbard meeting Erickson but Hubbard and Erickson were both highly active at the same time in Arizona and I think Hubbard must have sat in on Erickson’s lectures.) (That is, that weird ramblely kind of writing is by no means innocent but it is misparsed by many people's minds and helps create an altered state of consciousness.) Hubbard's most satanic (in opposition to conventional Eastern & Western religious ideas) works are Introduction to Scientology Ethics and A New Slant on Life. The first one describes an elaborate system of punishments that are applied to anyone who violates group norms and a system of evaluation that guarantees you will violate those norms by accident. The list of "ethics conditions" is folded so the positive conditions come first and a person who starts reading at the beginning and doesn't go all the way through will miss the batshit crazy stuff. The second describes how a Scientologist is supposed to disconnect themselves from anyone who disagrees with their spiritual journey. Something Scientology shares with the EY cult (as well as the LaRouche organization and the third international) is the proliferation of front groups. Scientology has Narconon, Criminon, The Way to Happiness Foundation, and many others. The EY cult has "effective altruism", "rationalism", "longtermism", "AI safety" and probably those parties Aella used to run. It's a typical path that a coercive group makes multiple 360° turns. "Effective Altruism" seems designed to attract morally weak people but seems rational enough at first. I mean, it seemed to make sense when Bill Gates insisted on funding only international NGOs that could prove the value of what they were doing. Once, however, you are funding "longtermist" organizations that are concerned with imagined problems on the way between here and the glorious future that we've turned all the planetary mass of the Milky Way into Dyson spheres there are no results to measure! It is like the technique a stage hypnotist uses of putting volunteers through a number of screens that rapidly selects the most suggestible and/or likely to go along subjects. Both in the short term and long term this process expels the trouble makers and leave behind the potential Manchurian Candidates.
- bnralt 4y ago
  I’ve noticed a similar writing style in a lot of others from the same group (AI Doomsdayers/“Rationalists”). Most of the time I stumble across something from that corner of the web it feels like the author was thinking “Why convey an idea in just one sentence when I can convey it in ten?”
  - deleted 4y ago
    [deleted]
  - xdavidliu 4y ago
    Reminds me a of a quote by Chomsky about post-modernism (not what EY is, but similar to what you're talking about) > Some of what appears in it sort of actually makes sense, but when you reproduce it in monosyllables, it turns out to be truisms. It’s perfectly true that when you look at scientists in the West, they’re mostly men, it’s perfectly true that women have had a hard time breaking into the scientific fields, and it’s perfectly true that there are institutional factors determining how science proceeds that reflect power structures. All of this can be described literally in monosyllables, and it turns out to be truisms. On the other hand, you don’t get to be a respected intellectual by presenting truisms in monosyllables. https://www.openculture.com/2013/07/noam-chomsky-calls-postmodern-critiques-of-science-over-inflated-polysyllabic-truisms.html https://www.openculture.com/2013/07/noam-chomsky-calls-postm... https://en.wikipedia.org/wiki/Obscurantism https://en.wikipedia.org/wiki/Obscurantism
  - PaulHoule 4y ago
    It's like Amway, people get into the EY cult and it takes over their life, they are trained to talk like EY to drive all the "normies" away, isolate them, and make them easier to control.
  - mitthrowaway2 4y ago
    Using a lot of words is often a response to getting repeatedly misinterpreted or misunderstood by others. It results in putting up ten explanatory, context, definitional, or guardrail sentences for each one sentence of novel content.
    - 4y ago
gpderetta 4y ago
Similarly on how people have fine tuned SD on Midjourney.
substation13 4y ago
Is it a good insight though? The training data for many of these premier models is already available, and that's even better than some input-output pairs.
- zarzavat 4y ago
  He’s talking about RLHF - reinforcement learning with human feedback (the process that trained ChatGPT), the training data for which is not publicly available. And the point is that you don’t need to RLHF as long as you have access to another model that has been trained with RLHF that you can blackbox.
  - substation13 4y ago
    I see, I didn't realize that training data for tuning was not available. Thanks!
sdwr 4y ago
That's only because these models are all pretty much the same (?) in terms of architecture, and basically do the same thing. New features, like memory, medium-term planning, and iterating towards a design goal, won't be imitatable in the same way.
yunohn 4y ago
> but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be Why does he use 2 different niche analogies to explain the thing that is still quite densely written at the end? Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…
- dragonwriter 4y ago
  > Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence… “Making profit on AI is basically the same as making profit on other software, except that the sensitive bits include the direct inputs and outputs to the core model, not just the software itself.” Yes, it could. But then, the attempt to sell the idea this was some kind of radical phase transition that the players in the AI industry (largely, very experienced players in the software industry) simply were fundamentally not prepared for would be a lot harder.

by cs702 4y ago

PaulHoule 4y ago

It’s not accidental at all that EY writes this way. He is writing out of L. Ron Hubbard’s style guide. He doesn’t want to communicate to outsiders, he is writing to members of his Cthulhu cult.

canadiantim 4y ago
Yeah I was liking his post and following along until he kept repeating "shoggoth", very weird. I stopped at that part
- deleted 4y ago
  [deleted]
- 4bpp 4y ago
  The "shoggoth with a smiley face" metaphor for LLMs has gained traction far beyond Yudkowsky's bubble. It even has a KYM article now: https://knowyourmeme.com/memes/shoggoth-with-smiley-face-artificial-intelligence https://knowyourmeme.com/memes/shoggoth-with-smiley-face-art...
  - semi-extrinsic 4y ago
    Doesn't make any of this less stupid though.
    - chpatrick 4y ago
      Makes sense to me. It's an enormous inhuman blob trained on all of human knowledge that can manifest a personality out of nothing based on some inoffensive corporate prompt.
jerry1979 4y ago
What do you mean by "L. Ron Hubbard's style guide"? Would that be doing things like employing in-group metaphors?
- PaulHoule 4y ago
  Making up lots of vocabulary that divides insiders from outsiders. I was interested in Scientology as a microcosm of society around the time alt.religion.scientology appeared in the early 1990s. Scientology documented everything about how it runs, even management is scriptural, and only Hubbard could write scripture so it is like a specimen that's been fixed to a microscope slide. Since then renegade "Free Zone" Scientologists have put all of his writings online so it is very accessible. Dianetics is the Sequences of Scientology in that it is a long, windy, and nonsensical book that selects for people who are perseverant readers who are impervious to critical thinking. (Like one of those Nigerian scam emails.) Hubbard's hypnotic language is demonstrated in his Philadelphia Doctorate Lectures where he not only mentions the hypnotist Milton Erickson but uses Erickson's techniques extensively. (There is no record of Hubbard meeting Erickson but Hubbard and Erickson were both highly active at the same time in Arizona and I think Hubbard must have sat in on Erickson’s lectures.) (That is, that weird ramblely kind of writing is by no means innocent but it is misparsed by many people's minds and helps create an altered state of consciousness.) Hubbard's most satanic (in opposition to conventional Eastern & Western religious ideas) works are Introduction to Scientology Ethics and A New Slant on Life. The first one describes an elaborate system of punishments that are applied to anyone who violates group norms and a system of evaluation that guarantees you will violate those norms by accident. The list of "ethics conditions" is folded so the positive conditions come first and a person who starts reading at the beginning and doesn't go all the way through will miss the batshit crazy stuff. The second describes how a Scientologist is supposed to disconnect themselves from anyone who disagrees with their spiritual journey. Something Scientology shares with the EY cult (as well as the LaRouche organization and the third international) is the proliferation of front groups. Scientology has Narconon, Criminon, The Way to Happiness Foundation, and many others. The EY cult has "effective altruism", "rationalism", "longtermism", "AI safety" and probably those parties Aella used to run. It's a typical path that a coercive group makes multiple 360° turns. "Effective Altruism" seems designed to attract morally weak people but seems rational enough at first. I mean, it seemed to make sense when Bill Gates insisted on funding only international NGOs that could prove the value of what they were doing. Once, however, you are funding "longtermist" organizations that are concerned with imagined problems on the way between here and the glorious future that we've turned all the planetary mass of the Milky Way into Dyson spheres there are no results to measure! It is like the technique a stage hypnotist uses of putting volunteers through a number of screens that rapidly selects the most suggestible and/or likely to go along subjects. Both in the short term and long term this process expels the trouble makers and leave behind the potential Manchurian Candidates.
bnralt 4y ago
I’ve noticed a similar writing style in a lot of others from the same group (AI Doomsdayers/“Rationalists”). Most of the time I stumble across something from that corner of the web it feels like the author was thinking “Why convey an idea in just one sentence when I can convey it in ten?”
- deleted 4y ago
  [deleted]
- xdavidliu 4y ago
  Reminds me a of a quote by Chomsky about post-modernism (not what EY is, but similar to what you're talking about) > Some of what appears in it sort of actually makes sense, but when you reproduce it in monosyllables, it turns out to be truisms. It’s perfectly true that when you look at scientists in the West, they’re mostly men, it’s perfectly true that women have had a hard time breaking into the scientific fields, and it’s perfectly true that there are institutional factors determining how science proceeds that reflect power structures. All of this can be described literally in monosyllables, and it turns out to be truisms. On the other hand, you don’t get to be a respected intellectual by presenting truisms in monosyllables. https://www.openculture.com/2013/07/noam-chomsky-calls-postmodern-critiques-of-science-over-inflated-polysyllabic-truisms.html https://www.openculture.com/2013/07/noam-chomsky-calls-postm... https://en.wikipedia.org/wiki/Obscurantism https://en.wikipedia.org/wiki/Obscurantism
- PaulHoule 4y ago
  It's like Amway, people get into the EY cult and it takes over their life, they are trained to talk like EY to drive all the "normies" away, isolate them, and make them easier to control.
- mitthrowaway2 4y ago
  Using a lot of words is often a response to getting repeatedly misinterpreted or misunderstood by others. It results in putting up ten explanatory, context, definitional, or guardrail sentences for each one sentence of novel content.
  - 4y ago

gpderetta 4y ago

Similarly on how people have fine tuned SD on Midjourney.

substation13 4y ago

Is it a good insight though? The training data for many of these premier models is already available, and that's even better than some input-output pairs.

zarzavat 4y ago
He’s talking about RLHF - reinforcement learning with human feedback (the process that trained ChatGPT), the training data for which is not publicly available. And the point is that you don’t need to RLHF as long as you have access to another model that has been trained with RLHF that you can blackbox.
- substation13 4y ago
  I see, I didn't realize that training data for tuning was not available. Thanks!

sdwr 4y ago

That's only because these models are all pretty much the same (?) in terms of architecture, and basically do the same thing. New features, like memory, medium-term planning, and iterating towards a design goal, won't be imitatable in the same way.

yunohn 4y ago

> but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be Why does he use 2 different niche analogies to explain the thing that is still quite densely written at the end? Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…

dragonwriter 4y ago
> Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence… “Making profit on AI is basically the same as making profit on other software, except that the sensitive bits include the direct inputs and outputs to the core model, not just the software itself.” Yes, it could. But then, the attempt to sell the idea this was some kind of radical phase transition that the players in the AI industry (largely, very experienced players in the software industry) simply were fundamentally not prepared for would be a lot harder.