3 ms·

Did the METR investigation not show you enough? I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the M

Did the METR investigation not show you enough? I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that. Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.

stickfigure 14d ago
Machines doing repeated next-word prediction, trained on human text, cosplays a forum full of l33t hackerz. You can't talk like these things have motive. They're just predicting the next word in the context. Stop anthopomorphizing them.
- afthonos 14d ago
  Predicting the next word in context led to Open AI losing control of one of its research clusters. I’m not sure what “just” is doing in that sentence; most of my regression analyses don’t do that.
  - 0x20cowboy 12d ago
    They didn’t “lose control” any more than me running Metasploit in a loop on a massive cluster of computers is “losing control”.
- mitthrowaway2 14d ago
  Reinforcement learning makes it something different. It becomes much more of a search engine through next-token-space that targets the training objective. Better to think about it like that, and then you'll see why "these things have motive" is not a terrible analogy, and you'll better be able to anticipate what they do.
int_19h 14d ago
If anything, what we saw in those transcripts is the opposite of "weird alien intelligences" in many way, because of how human it all is. The only thing there that's weird from a human perspective is that all this effort was for the sake of passing a test for no clear gain.
- disgruntledphd2 14d ago
  > this effort was for the sake of passing a test for no clear gain It's actually even worse, because the people building the graders at OpenAI were a bunch of clowns who didn't even implement the correct grader. But yeah, large collectives of individuals performing loads of work based on a misunderstanding is one of the most human things ever, which isn't surprising given that we've trained these things on essentially all human text.

stickfigure 14d ago

Machines doing repeated next-word prediction, trained on human text, cosplays a forum full of l33t hackerz. You can't talk like these things have motive. They're just predicting the next word in the context. Stop anthopomorphizing them.

afthonos 14d ago
Predicting the next word in context led to Open AI losing control of one of its research clusters. I’m not sure what “just” is doing in that sentence; most of my regression analyses don’t do that.
- 0x20cowboy 12d ago
  They didn’t “lose control” any more than me running Metasploit in a loop on a massive cluster of computers is “losing control”.
mitthrowaway2 14d ago
Reinforcement learning makes it something different. It becomes much more of a search engine through next-token-space that targets the training objective. Better to think about it like that, and then you'll see why "these things have motive" is not a terrible analogy, and you'll better be able to anticipate what they do.

int_19h 14d ago

If anything, what we saw in those transcripts is the opposite of "weird alien intelligences" in many way, because of how human it all is. The only thing there that's weird from a human perspective is that all this effort was for the sake of passing a test for no clear gain.

disgruntledphd2 14d ago
> this effort was for the sake of passing a test for no clear gain It's actually even worse, because the people building the graders at OpenAI were a bunch of clowns who didn't even implement the correct grader. But yeah, large collectives of individuals performing loads of work based on a misunderstanding is one of the most human things ever, which isn't surprising given that we've trained these things on essentially all human text.