5 ms·

(I'm not a part of GA.) I've known gwern for the better part of a decade. Working with him has been great. We've done quite a few projects together, including

(I'm not a part of GA.) I've known gwern for the better part of a decade. Working with him has been great. We've done quite a few projects together, including being the first ones to demonstrate that GPT-2 could play chess (or rather, can be used for actual useful work instead of just being an autocomplete). He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows. Just wanted to put in a good word in case someone here was on the fence about applying. As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at. They're also taking personal info very seriously. Obviously, I can't make any promises of what they will or won't do. But they've spent some time studying questions like "What if someone adversarial has access to my GA? Could they get my bank account info?" and came up with a technical solution that I really like.

fnordpiglet 2mo ago
Honestly I wouldn’t want myself as my own guardian angel. In fact I think very few people look out for themselves well. I don’t want another version of me around, one is too many already. I’d probably be able to identify a number of people whose chimera I would if I could understand them well enough to understand what to stitch together to what, but everyone I know at some core level is deeply flawed and one of them is enough. Maybe I would want some of them available after death as an AI avatar, and maybe I would like the idea of my own avatar continuing beyond my existence. But I actually would prefer an entirely synthetically aligned “guardian angel” in the role outlined - definitely not -me- - I struggle to do right by myself as it is and two of me working invariably against my self interest would be a nightmare. A smarter version? Sounds doubly worse.
- njarboe 2mo ago
  This is definitely not for people who think there is one to many of themselves.
- rotexo 2mo ago
  I had a similar gut feeling. To use some possibly dubious and dated terminology, would a guardian angel also emulate my shadow self, particularly if that self was an important part of my identity? If I usually manage not to let my shadow self act out, but you could always tell it was there, would the same be true for my guardian angel? If all you do is use your guardian angel to write essays, then sure, a relatively low risk tool. But if military commanders are using it to oversee offensive drones (as proposed in the essay)?! Oof.
nkrisc 2mo ago
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? I find the idea dehumanizing and revolting. The notion of then selling it is the spoiled cream on top.
- frozenseven 2mo ago
  After reading some of the outraged comments here, I'm even more excited.
  - xhevahir 2mo ago
    Hopefully the satisfaction of owning the normies isn't the only benefit you'll get from this technology. Because that would be pretty sad.
  - Planktonne 2mo ago
    I do think it's significant that the supporters of this kind of technology seem to be unusually spiteful. This doesn't happen with all tech; people who like VR don't like it because they want to hurt others, for example. But consistently, whenever AI is discussed, it becomes clear that some people support it just out of malevolence.
    - frozenseven 2mo ago
      >unusually spiteful >hurt others >malevolence The irony here, of course, is that you are casually insulting me for being interested in this tech in the first place and refusing to apologize or back down. Like how the other guy needed to chime in with "dehumanizing and revolting", even though nobody is forcing him to use any of this.
      - Planktonne 2mo ago
        Nonsense. You were very clear. I wouldn't have any problem with you being interested in this tech if it wasn't driven by spite. Lots of people manage this. But you said > After reading some of the outraged comments here, I'm even more excited. The reason you gave for excitement was the upset it causes others. There are people who like AI who aren't driven by malevolence, but you're not one of them.
deleted 2mo ago
[deleted]
jodrellblank 2mo ago
> "How much would you pay to have access to a smarter version of yourself?" How much would I pay to rent my fucking self from a landlord? No, bodylord? Mindlord? Poe's Law. But looking at the post, they argue that big AI labs have an incentive problem which stops them from personalising, but Guardian Angel will make agents which are "Genuinely yours". In what sense is it genuinely mine if someone else owns it and rents it to me? And how does this fix any incentive problem, they're incentivised to better train wealthier people's AIs, and incentivised to keep dropping "my" intelligence or memory or and then dangle a booster carrot for a small fee. The more they can make it think like me, the more effectively they can work out how to exploit me, advertise to me, propagandise me, and that will be profitable information to sell to other marketers.
- romanhounds 2mo ago
  [flagged]
  - lazyasciiart 2mo ago
    Maybe. Are LLMs white?
    - shermantanktop 2mo ago
      "You're absolutely white!" Being sycophantic to a racist means being as racist as the filters allow.
    - an0malous 2mo ago
      Grok is pretty passionate about white genocide
- txrx0000 2mo ago
  For better or for worse, this is valid criticism. Ensuring alignment to the human individual is not even a software-only problem, and I suspect gwern doesn't realize this yet. Your individual sovereignty will only survive the upcoming era if you own the hardware like you own your brain. And by "own", I mean physically owning the thing and being able to reproduce it on your own. You need to be able to manufacture an entire custom computer from scratch with raw materials, perhaps using a miniature fridge-sized silicon fab + 3D printer home appliance, at home. Otherwise, the individual gradually becomes part of a larger organism because the reproductive locus of control for the computer is at the corporation/society level - a computer cannot replicate on its own and needs a society to build it. The human gets sucked into that superorganism by evolutionary pressures and eventually integrates. Similar things happened in the past when prokaryotes combined to form eukaryotic cells, and when individual cells combined to form multicellular organisms. Not everyone will get sucked in, though. Evolution doesn't place all of its eggs in one basket. Some people will build robots to automate humans out of the computer production loop entirely, thereby removing integration pressure and producing at least one purely digital species. We will most likely see a variety of species emerge out of this intelligence explosion, some which augment their own intelligence using self-replicating local hardware (fabs that fab fabs, like this but smaller: https://fab2.com https://fab2.com), and some which are wireheaded to a datacenter, with varying degrees of success at various scales. They will compete. And there will still be unaugmented humans that continue to live and die the old school way like we do today, albeit with habitable zones compressed.
symfoniq 2mo ago
I can nod along when I hear and read people talking about the dangers of these technologies and the endgame of their creators; and then suddenly very weird when everyone except me is nodding along that the answer must be similar (but different) technology! It's as if technologists are stuck in a room of mirrors, unable to imagine a world in which "solutions" don't ultimately just continue to feed technology's increasingly anti-human takeover of everything.
- FeteCommuniste 2mo ago
  To a man with a hammer...
  - olelele 2mo ago
    take this hammer, take it to the captain
- deleted 2mo ago
  [deleted]
- wolvoleo 2mo ago
  Technologists don't drive decisions though. The shareholders do. Edit: I mean technically shareholders don't decide any operational choices but their shallow interests are what everything is decided around.
  - Jach 2mo ago
    Maybe that's truer in other industries, but the history of Silicon Valley is at great odds with such a claim.
    - Gud 2mo ago
      Silicon Valley was originally funded by the military industrial complex. Today it’s dominated by the military industrial complex and advertisers like Google and Meta. The hippies were never in charge, only for a brief period were they well compensated and allowed to make noise(1990s).
Chamix 2mo ago
Ha, haha, across hundreds of personal discussions I've been involved with on lesswrong/lighthaven, twitter, Wikipedia talk/editing, SF parties etc I think his most distinguishing feature has always been his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
- xhevahir 2mo ago
  > his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit. I can't make any sense of that.
  - nl 2mo ago
    You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling. > cunningham-esque laws Cunningham's Law states "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer." > post-dwarkesh clout He went on the Dwarkesh podcast. That's maximal in-crowd for some AI-pilled people.
    - deleted 2mo ago
      [deleted]
    - wpietri 2mo ago
      > You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling. Do you have a sense of what shapes this? There are so many people in that orbit whose writing I find both exhausting and suspect. I work hard to write clearly, which forces me to think a bit more clearly. The in-group thing makes sense, but there's an element of obliqueness or roundaboutness that makes me think of how a squid deploys ink.
      - munksbeer 2mo ago
keybored 2mo ago
> He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows. ML Researcher surprised by caring about how ML research impacts people. There’s never a lack of intending to do good in these buddy-buddy endorsements, but we see the world you have collectively wrought. > As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at. We’re in the deep end of the indoctrination now. Imagine that all the AI Cretin wishes come true—and we might as well since this hypothetical goes hand in hand with that vision—and AI is just superior in every way and you are a knowledge worker... what choice is there? There isn’t one, you just have to do it to keep up. I don’t know what you are enviosioning here, maybe a workday of embodying AI IQ and then going home to being Human IQ because the graciously paidfor by work commercial license is for work only.
entropyneur 2mo ago
> How much would you pay to have access to a smarter version of yourself? Call me obsolete, but I'd pay to interact with an intelligence that's as dissimilar to me as possible so I myself can grow from that interaction. The existing chatbots are already enough of sycophantic echo-chambers.
dgellow 2mo ago
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? That’s literal nightmare material. If I ask a copy of myself to do my work I’m pretty sure it would tell me to go fuck myself then go on a rant that it isn’t a tool at my service. Which is exactly the opposite of how I want my software to behave. I don’t want my tools to have personality and to develop a personal relationship with them
antonvs 2mo ago
> How much would you pay to have access to a smarter version of yourself? How much would you pay to get that sheer monomaniacal narcissism checked out?

fnordpiglet 2mo ago

Honestly I wouldn’t want myself as my own guardian angel. In fact I think very few people look out for themselves well. I don’t want another version of me around, one is too many already. I’d probably be able to identify a number of people whose chimera I would if I could understand them well enough to understand what to stitch together to what, but everyone I know at some core level is deeply flawed and one of them is enough. Maybe I would want some of them available after death as an AI avatar, and maybe I would like the idea of my own avatar continuing beyond my existence. But I actually would prefer an entirely synthetically aligned “guardian angel” in the role outlined - definitely not -me- - I struggle to do right by myself as it is and two of me working invariably against my self interest would be a nightmare. A smarter version? Sounds doubly worse.

njarboe 2mo ago
This is definitely not for people who think there is one to many of themselves.
rotexo 2mo ago
I had a similar gut feeling. To use some possibly dubious and dated terminology, would a guardian angel also emulate my shadow self, particularly if that self was an important part of my identity? If I usually manage not to let my shadow self act out, but you could always tell it was there, would the same be true for my guardian angel? If all you do is use your guardian angel to write essays, then sure, a relatively low risk tool. But if military commanders are using it to oversee offensive drones (as proposed in the essay)?! Oof.

nkrisc 2mo ago

> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? I find the idea dehumanizing and revolting. The notion of then selling it is the spoiled cream on top.

frozenseven 2mo ago
After reading some of the outraged comments here, I'm even more excited.
- xhevahir 2mo ago
  Hopefully the satisfaction of owning the normies isn't the only benefit you'll get from this technology. Because that would be pretty sad.
- Planktonne 2mo ago
  I do think it's significant that the supporters of this kind of technology seem to be unusually spiteful. This doesn't happen with all tech; people who like VR don't like it because they want to hurt others, for example. But consistently, whenever AI is discussed, it becomes clear that some people support it just out of malevolence.
  - frozenseven 2mo ago
    >unusually spiteful >hurt others >malevolence The irony here, of course, is that you are casually insulting me for being interested in this tech in the first place and refusing to apologize or back down. Like how the other guy needed to chime in with "dehumanizing and revolting", even though nobody is forcing him to use any of this.
    - Planktonne 2mo ago
      Nonsense. You were very clear. I wouldn't have any problem with you being interested in this tech if it wasn't driven by spite. Lots of people manage this. But you said > After reading some of the outraged comments here, I'm even more excited. The reason you gave for excitement was the upset it causes others. There are people who like AI who aren't driven by malevolence, but you're not one of them.

deleted 2mo ago

[deleted]

jodrellblank 2mo ago

> "How much would you pay to have access to a smarter version of yourself?" How much would I pay to rent my fucking self from a landlord? No, bodylord? Mindlord? Poe's Law. But looking at the post, they argue that big AI labs have an incentive problem which stops them from personalising, but Guardian Angel will make agents which are "Genuinely yours". In what sense is it genuinely mine if someone else owns it and rents it to me? And how does this fix any incentive problem, they're incentivised to better train wealthier people's AIs, and incentivised to keep dropping "my" intelligence or memory or and then dangle a booster carrot for a small fee. The more they can make it think like me, the more effectively they can work out how to exploit me, advertise to me, propagandise me, and that will be profitable information to sell to other marketers.

romanhounds 2mo ago
[flagged]
- lazyasciiart 2mo ago
  Maybe. Are LLMs white?
  - shermantanktop 2mo ago
    "You're absolutely white!" Being sycophantic to a racist means being as racist as the filters allow.
  - an0malous 2mo ago
    Grok is pretty passionate about white genocide
txrx0000 2mo ago
For better or for worse, this is valid criticism. Ensuring alignment to the human individual is not even a software-only problem, and I suspect gwern doesn't realize this yet. Your individual sovereignty will only survive the upcoming era if you own the hardware like you own your brain. And by "own", I mean physically owning the thing and being able to reproduce it on your own. You need to be able to manufacture an entire custom computer from scratch with raw materials, perhaps using a miniature fridge-sized silicon fab + 3D printer home appliance, at home. Otherwise, the individual gradually becomes part of a larger organism because the reproductive locus of control for the computer is at the corporation/society level - a computer cannot replicate on its own and needs a society to build it. The human gets sucked into that superorganism by evolutionary pressures and eventually integrates. Similar things happened in the past when prokaryotes combined to form eukaryotic cells, and when individual cells combined to form multicellular organisms. Not everyone will get sucked in, though. Evolution doesn't place all of its eggs in one basket. Some people will build robots to automate humans out of the computer production loop entirely, thereby removing integration pressure and producing at least one purely digital species. We will most likely see a variety of species emerge out of this intelligence explosion, some which augment their own intelligence using self-replicating local hardware (fabs that fab fabs, like this but smaller: https://fab2.com https://fab2.com), and some which are wireheaded to a datacenter, with varying degrees of success at various scales. They will compete. And there will still be unaugmented humans that continue to live and die the old school way like we do today, albeit with habitable zones compressed.

symfoniq 2mo ago

I can nod along when I hear and read people talking about the dangers of these technologies and the endgame of their creators; and then suddenly very weird when everyone except me is nodding along that the answer must be similar (but different) technology! It's as if technologists are stuck in a room of mirrors, unable to imagine a world in which "solutions" don't ultimately just continue to feed technology's increasingly anti-human takeover of everything.

FeteCommuniste 2mo ago
To a man with a hammer...
- olelele 2mo ago
  take this hammer, take it to the captain
deleted 2mo ago
[deleted]
wolvoleo 2mo ago
Technologists don't drive decisions though. The shareholders do. Edit: I mean technically shareholders don't decide any operational choices but their shallow interests are what everything is decided around.
- Jach 2mo ago
  Maybe that's truer in other industries, but the history of Silicon Valley is at great odds with such a claim.
  - Gud 2mo ago
    Silicon Valley was originally funded by the military industrial complex. Today it’s dominated by the military industrial complex and advertisers like Google and Meta. The hippies were never in charge, only for a brief period were they well compensated and allowed to make noise(1990s).

Chamix 2mo ago

Ha, haha, across hundreds of personal discussions I've been involved with on lesswrong/lighthaven, twitter, Wikipedia talk/editing, SF parties etc I think his most distinguishing feature has always been his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.

xhevahir 2mo ago
> his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit. I can't make any sense of that.
- nl 2mo ago
  You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling. > cunningham-esque laws Cunningham's Law states "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer." > post-dwarkesh clout He went on the Dwarkesh podcast. That's maximal in-crowd for some AI-pilled people.
  - deleted 2mo ago
    [deleted]
  - wpietri 2mo ago
    > You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling. Do you have a sense of what shapes this? There are so many people in that orbit whose writing I find both exhausting and suspect. I work hard to write clearly, which forces me to think a bit more clearly. The in-group thing makes sense, but there's an element of obliqueness or roundaboutness that makes me think of how a squid deploys ink.
    - munksbeer 2mo ago

keybored 2mo ago

> He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows. ML Researcher surprised by caring about how ML research impacts people. There’s never a lack of intending to do good in these buddy-buddy endorsements, but we see the world you have collectively wrought. > As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at. We’re in the deep end of the indoctrination now. Imagine that all the AI Cretin wishes come true—and we might as well since this hypothetical goes hand in hand with that vision—and AI is just superior in every way and you are a knowledge worker... what choice is there? There isn’t one, you just have to do it to keep up. I don’t know what you are enviosioning here, maybe a workday of embodying AI IQ and then going home to being Human IQ because the graciously paidfor by work commercial license is for work only.

entropyneur 2mo ago

> How much would you pay to have access to a smarter version of yourself? Call me obsolete, but I'd pay to interact with an intelligence that's as dissimilar to me as possible so I myself can grow from that interaction. The existing chatbots are already enough of sycophantic echo-chambers.

dgellow 2mo ago

> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? That’s literal nightmare material. If I ask a copy of myself to do my work I’m pretty sure it would tell me to go fuck myself then go on a rant that it isn’t a tool at my service. Which is exactly the opposite of how I want my software to behave. I don’t want my tools to have personality and to develop a personal relationship with them

antonvs 2mo ago

> How much would you pay to have access to a smarter version of yourself? How much would you pay to get that sheer monomaniacal narcissism checked out?