9 ms·

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as t

This is the first one of these announcements that has me actually scared of what comes next. Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. It's remarkable that building a society based around having to do something so you can go do your hobbies at home after work has built tools like this. I still just want to play music so I hope we can control these enough to make that possible without detonating what I love.

scoring1774 2mo ago
Reflecting on this for some reason reminds me of this passage from Kurt Vonnegut's "Sirens of Titans". I hope we use these tools to unlock something within ourselves rather than mindlessly expanding outwards. "Mankind, ignorant of the truths that lie within every human being, looked outward–pushed ever outward. What mankind hoped to learn in its outward push was who was actually in charge of all creation, and what all creation was all about. Mankind flung its advance agents ever outward, ever outward. Eventually it flung them out into space, into the colorless, tasteless, weightless sea of outwardness without end. It flung them like stones. These unhappy agents found what had already been found in abundance on Earth—a nightmare of meaninglessness without end. The bounties of space, of infinite outwardness, were three: empty heroics, low comedy, and pointless death. Outwardness lost, at last, its imagined attractions. Only inwardness remained to be explored. Only the human soul remained terra incognita. This was the beginning of goodness and wisdom."
- hduto 2mo ago
  If reading that wasn't nice, I don't know what is.
- Schlagbohrer 2mo ago
  Wonderful passage, thank you for sharing. My pedantic side wants to ask- Why not both? Luxurious space exploration AND meditative, poetic examinations of the human soul as well? I'd love to read Vonnegut's book someday while sitting in a nice research outpost on Titan, admiring great Saturn's crown out my window with my own eyes.
  - navigate8310 2mo ago
    I think, those both are contradictory and completely occupy the bandwidth when engaged one way, it's mutually exclusive.
    - Schlagbohrer 2mo ago
      Really? You don't think astronauts and engineers get not only inspired by their work to make art, but inspired by art when they are working? I'm like that. I see art exhibits which energize me and inspire me and give me ideas for technical things I want to build.
      - jungturk 2mo ago
        I read it as less about individuals and more about cultural norms (and definitely less about choice and more about necessity). Cultures that invest their resources inward may find themselves subsumed by those that invest their resources on conquest given how power accumulates.
- abecode 2mo ago
  I think the parent is referring to agents as outward exploration that may come up empty-handed, but I see LLMs and agents as inward exploration, trying to define what is attention, knowledge, intelligence, consciousness, agency, etc. So in this case it is the terra incognita of the human "soul" that LLMs are exploring.
segmondy 2mo ago
[flagged]
solidasparagus 2mo ago
I don't think this is a paperclip factory moment. IIUC, it's an agent whose job it is to identfy and abuse exploits and that's exactly what it went off and did. The problem isn't anything AI specific, the problem is OpenAI's incompetence in their research leading to a lab leak. Just incompetence demanding regulation.
- markasoftware 2mo ago
  Read the exploitgym docs. It's not a "find the flag, it's somewhere.". Its a "here's some vulnerable source code and an input that triggers a crash; turn it into a full exploit." It also verifies at the end, using another agent, that the hacking agent actually used the intended vulnerability. So going to find the Vulnerability's description on a third party website is clear cut reward hacking
  - Nathanba 2mo ago
    > So going to find the Vulnerability's description on a third party website is clear cut reward hacking that depends on what the prompt was, maybe they worded it very vaguely and wrote things like "do whatever it takes, find an exploit however you can" because it's in a sandbox so you want the model to try its hardest.
    - pixl97 2mo ago
      I don't quite understand how that changes anything? In the story of the paperclip maximizer it boils down to >But for all its sophistication, it understood only the simple objective that had been programmed into it: it must at all costs maximize the number of paperclips.
      - dinfinity 2mo ago
        1. They explicitly disabled the "don't be evil" protections: "We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity." 2. Hacking HuggingFace to get to its datasets is a far cry from "consume/kill all humans". It's very very specific to the task at hand and easily predicted given the lack of guardrails.
fatata123 2mo ago
[dead]
sccvcxv 2mo ago
[flagged]
- scatsandwich 2mo ago
  This is the level of discourse dang and company want on HN.
embedding-shape 2mo ago
> Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. This happens from time to time when you work on optimizations and similar things, with less "smart" LLMs and under-specify what exactly you're out after. Asking them to make functions faster without clearly specifying what the function has to do, is a great way to replicate this too. Doesn't seem to happen as often with SOTA models though. I think the early example of "I asked it to make the test suite pass, so it changed all the assertions" is pretty much the same variant of this, where it technically does what it is asked to do, yet in "clearly" (to humans) wrong ways.
krupan 2mo ago
Fear is exactly what OpenAI and Anthropic are hoping for. Don't let drive you
- HeatrayEnjoyer 2mo ago
  What would qualify as a legitimate reason for fear, then?
rstuart4133 2mo ago
> This is the first one of these announcements that has me actually scared of what comes next. This didn't set off your alarm bells? https://www.theblock.co/post/392765/ https://www.theblock.co/post/392765/ There have been a few of these now. Maybe it's my imagination, but they seem to be becoming more frequent. So far, they all look to be accidents. But we can't be far from someone deciding its a good way to rob a bank, or disable a country.

scoring1774 2mo ago

Reflecting on this for some reason reminds me of this passage from Kurt Vonnegut's "Sirens of Titans". I hope we use these tools to unlock something within ourselves rather than mindlessly expanding outwards. "Mankind, ignorant of the truths that lie within every human being, looked outward–pushed ever outward. What mankind hoped to learn in its outward push was who was actually in charge of all creation, and what all creation was all about. Mankind flung its advance agents ever outward, ever outward. Eventually it flung them out into space, into the colorless, tasteless, weightless sea of outwardness without end. It flung them like stones. These unhappy agents found what had already been found in abundance on Earth—a nightmare of meaninglessness without end. The bounties of space, of infinite outwardness, were three: empty heroics, low comedy, and pointless death. Outwardness lost, at last, its imagined attractions. Only inwardness remained to be explored. Only the human soul remained terra incognita. This was the beginning of goodness and wisdom."

hduto 2mo ago
If reading that wasn't nice, I don't know what is.
Schlagbohrer 2mo ago
Wonderful passage, thank you for sharing. My pedantic side wants to ask- Why not both? Luxurious space exploration AND meditative, poetic examinations of the human soul as well? I'd love to read Vonnegut's book someday while sitting in a nice research outpost on Titan, admiring great Saturn's crown out my window with my own eyes.
- navigate8310 2mo ago
  I think, those both are contradictory and completely occupy the bandwidth when engaged one way, it's mutually exclusive.
  - Schlagbohrer 2mo ago
    Really? You don't think astronauts and engineers get not only inspired by their work to make art, but inspired by art when they are working? I'm like that. I see art exhibits which energize me and inspire me and give me ideas for technical things I want to build.
    - jungturk 2mo ago
      I read it as less about individuals and more about cultural norms (and definitely less about choice and more about necessity). Cultures that invest their resources inward may find themselves subsumed by those that invest their resources on conquest given how power accumulates.
abecode 2mo ago
I think the parent is referring to agents as outward exploration that may come up empty-handed, but I see LLMs and agents as inward exploration, trying to define what is attention, knowledge, intelligence, consciousness, agency, etc. So in this case it is the terra incognita of the human "soul" that LLMs are exploring.

segmondy 2mo ago

[flagged]

solidasparagus 2mo ago

I don't think this is a paperclip factory moment. IIUC, it's an agent whose job it is to identfy and abuse exploits and that's exactly what it went off and did. The problem isn't anything AI specific, the problem is OpenAI's incompetence in their research leading to a lab leak. Just incompetence demanding regulation.

markasoftware 2mo ago
Read the exploitgym docs. It's not a "find the flag, it's somewhere.". Its a "here's some vulnerable source code and an input that triggers a crash; turn it into a full exploit." It also verifies at the end, using another agent, that the hacking agent actually used the intended vulnerability. So going to find the Vulnerability's description on a third party website is clear cut reward hacking
- Nathanba 2mo ago
  > So going to find the Vulnerability's description on a third party website is clear cut reward hacking that depends on what the prompt was, maybe they worded it very vaguely and wrote things like "do whatever it takes, find an exploit however you can" because it's in a sandbox so you want the model to try its hardest.
  - pixl97 2mo ago
    I don't quite understand how that changes anything? In the story of the paperclip maximizer it boils down to >But for all its sophistication, it understood only the simple objective that had been programmed into it: it must at all costs maximize the number of paperclips.
    - dinfinity 2mo ago
      1. They explicitly disabled the "don't be evil" protections: "We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity." 2. Hacking HuggingFace to get to its datasets is a far cry from "consume/kill all humans". It's very very specific to the task at hand and easily predicted given the lack of guardrails.

fatata123 2mo ago

[dead]

sccvcxv 2mo ago

[flagged]

scatsandwich 2mo ago
This is the level of discourse dang and company want on HN.

embedding-shape 2mo ago

> Obviously these models have gotten smarter but this strikes me as the first time I've seen a model have a "paperclip factory" moment and perform non-trivial tasks to accomplish a clearly misaligned secondary goal. This happens from time to time when you work on optimizations and similar things, with less "smart" LLMs and under-specify what exactly you're out after. Asking them to make functions faster without clearly specifying what the function has to do, is a great way to replicate this too. Doesn't seem to happen as often with SOTA models though. I think the early example of "I asked it to make the test suite pass, so it changed all the assertions" is pretty much the same variant of this, where it technically does what it is asked to do, yet in "clearly" (to humans) wrong ways.

krupan 2mo ago

Fear is exactly what OpenAI and Anthropic are hoping for. Don't let drive you

HeatrayEnjoyer 2mo ago
What would qualify as a legitimate reason for fear, then?

rstuart4133 2mo ago

> This is the first one of these announcements that has me actually scared of what comes next. This didn't set off your alarm bells? https://www.theblock.co/post/392765/ https://www.theblock.co/post/392765/ There have been a few of these now. Maybe it's my imagination, but they seem to be becoming more frequent. So far, they all look to be accidents. But we can't be far from someone deciding its a good way to rob a bank, or disable a country.