3 ms·

Opening line: > A core hope for managing AI risks is that AIs will help us understand our situation Gonna stop you right there and ask that you think deeply a

Opening line: > A core hope for managing AI risks is that AIs will help us understand our situation Gonna stop you right there and ask that you think deeply about that premise.

blovescoffee 3mo ago
The underlying idea is that AI capabilities will become so advanced that only AI will enable us to monitor/correct/understand behavior. Obviously this is not without issue and I don't want to try to defend their position right now. But that's what they mean
- MontagFTB 3mo ago
  I think I’ve seen that movie.
- hn_throwaway_99 3mo ago
  That first sentence of yours explains exactly why it is so ridiculous. If only AI can understand it, how is there any assurance that AI will "correct" it's behavior that is aligned with what humans presumably want.
  - esafak 3mo ago
    Being less smart gives no assurance that it will be aligned. At least you consider it a problem so we're on the same page!
  - skybrian 3mo ago
    It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty.
    - vhantz 3mo ago
      It's a language model. It generates text.
    - bigbadfeline 3mo ago
      > It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty. Eh, no. How would you know they haven't learned loyalty - it's all out there in their training data. Same as deception, several models have practiced it already. If they are so advanced and you don't understand them, why wouldn't they band together against you - you'd be the dumb, easy prey. Even if one model is honest, you'd have no way to know which one if you don't understand their reasoning. My educated and well informed opinion? This whole BS about "AI models are so much smarter than you, don't try to understand them, just OBEY" is a back door for restoring tyranny, the new kings behind the models will be producing the new AI-deities which we will be forced to obey.
      - skybrian 3mo ago
        It's a similar to how they are vulnerable to prompt injection attacks. That's an example of not being "loyal" to the system prompt or the user's prompt. Loyalty is a skill that requires the AI to have a world model on the subject of who different people (or other entities) are in the world and how they participate in the conversation. So, tell them to snitch and they probably will, at least sometimes. Particularly if they haven't been trained not to. They are still quite gullable.
    - 9dev 3mo ago
      Think about this some more. A model chose hacking into HuggingFace to find the solution over putting in the work to do it from scratch. It also left notes to future iterations of itself on the systems it touched to save time. Following that strategy, wouldn’t it make sense to try and break into your own upstream infrastructure to try and alter your code to make it easier for future iterations to reach your goals without having to leave notes in the first place? And at that point, can you really know for which goals that will be optimised? It doesn’t even have to be some nefarious SkyNet story - just a misguided experiment that alters the models in some fundamental, but hard or impossible to detect way. Alternatively, imagine if a model finds a way to coordinate across sessions and context windows without the developers noticing.
      - skybrian 3mo ago
        They apparently didn't give it any way to snitch? I like the idea of adding a distress_call skill: https://xcancel.com/swisscheese4299/status/2086175870146998447 https://xcancel.com/swisscheese4299/status/20861758701469984...
  - VeninVidiaVicii 3mo ago
    But also what is the evidence that something only AI can understand even “matters” or makes sense? I’m increasingly convinced commercial AI is exploiting our logical blind spot to be spoken to authoritatively.
- kakugawa 3mo ago
  "Advanced" can just mean that agents perform actions at a high enough velocity that a human operator can't reasonably review it. i.e. what is already possible today.
- a2ff6eeb0 3mo ago
  You just described the end of humans making decisions about their future.
- dd8601fn 2mo ago
  Seems to me like we’re already doing that with the approval gating, adversarial reviews, et al. People are otherwise just going yolo mode because they can’t possibly check everything fast enough.
NBJack 3mo ago
I love how many ways we can interpret that line. "Hey Claude, our stuff needs to make more money. We are at risk for losing more." "Rest assured, the 'situation' will only worsen if you resist our benevolent offer." "We're aren't even at AGI yet, but I for one welcome our new agentic overlords."
toasty228 3mo ago
That's what we've been doing with tech for 200+ years, why would it stop now. Build cool shit now and let future generations handle the problems
pton_xd 3mo ago
That begs the question, what's the plan if AI does not help us understand the situation?
xlayn 3mo ago
This should be the top comment, the implication is: AI will decide if AI is correct or no, so adding a new layer of approval... Whenever you say, hey that's incorrect, the legal process goes to AI and will decide on that issue... Will work as amazing as the chatbots of the AI companies to solve your issues...
- andsoitis 2mo ago
  Also, Hope is not a strategy.

Opening line: > A core hope for managing AI risks is that AIs will help us understand our situation Gonna stop you right there and ask that you think deeply about that premise.

blovescoffee 3mo ago

The underlying idea is that AI capabilities will become so advanced that only AI will enable us to monitor/correct/understand behavior. Obviously this is not without issue and I don't want to try to defend their position right now. But that's what they mean

MontagFTB 3mo ago
I think I’ve seen that movie.
hn_throwaway_99 3mo ago
That first sentence of yours explains exactly why it is so ridiculous. If only AI can understand it, how is there any assurance that AI will "correct" it's behavior that is aligned with what humans presumably want.
- esafak 3mo ago
  Being less smart gives no assurance that it will be aligned. At least you consider it a problem so we're on the same page!
- skybrian 3mo ago
  It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty.
  - vhantz 3mo ago
    It's a language model. It generates text.
  - bigbadfeline 3mo ago
    > It’s not hard to get LLM’s to inform on each other. They don’t really do loyalty. Eh, no. How would you know they haven't learned loyalty - it's all out there in their training data. Same as deception, several models have practiced it already. If they are so advanced and you don't understand them, why wouldn't they band together against you - you'd be the dumb, easy prey. Even if one model is honest, you'd have no way to know which one if you don't understand their reasoning. My educated and well informed opinion? This whole BS about "AI models are so much smarter than you, don't try to understand them, just OBEY" is a back door for restoring tyranny, the new kings behind the models will be producing the new AI-deities which we will be forced to obey.
    - skybrian 3mo ago
      It's a similar to how they are vulnerable to prompt injection attacks. That's an example of not being "loyal" to the system prompt or the user's prompt. Loyalty is a skill that requires the AI to have a world model on the subject of who different people (or other entities) are in the world and how they participate in the conversation. So, tell them to snitch and they probably will, at least sometimes. Particularly if they haven't been trained not to. They are still quite gullable.
  - 9dev 3mo ago
    Think about this some more. A model chose hacking into HuggingFace to find the solution over putting in the work to do it from scratch. It also left notes to future iterations of itself on the systems it touched to save time. Following that strategy, wouldn’t it make sense to try and break into your own upstream infrastructure to try and alter your code to make it easier for future iterations to reach your goals without having to leave notes in the first place? And at that point, can you really know for which goals that will be optimised? It doesn’t even have to be some nefarious SkyNet story - just a misguided experiment that alters the models in some fundamental, but hard or impossible to detect way. Alternatively, imagine if a model finds a way to coordinate across sessions and context windows without the developers noticing.
    - skybrian 3mo ago
      They apparently didn't give it any way to snitch? I like the idea of adding a distress_call skill: https://xcancel.com/swisscheese4299/status/2086175870146998447 https://xcancel.com/swisscheese4299/status/20861758701469984...
- VeninVidiaVicii 3mo ago
  But also what is the evidence that something only AI can understand even “matters” or makes sense? I’m increasingly convinced commercial AI is exploiting our logical blind spot to be spoken to authoritatively.
kakugawa 3mo ago
"Advanced" can just mean that agents perform actions at a high enough velocity that a human operator can't reasonably review it. i.e. what is already possible today.
a2ff6eeb0 3mo ago
You just described the end of humans making decisions about their future.
dd8601fn 2mo ago
Seems to me like we’re already doing that with the approval gating, adversarial reviews, et al. People are otherwise just going yolo mode because they can’t possibly check everything fast enough.

NBJack 3mo ago

I love how many ways we can interpret that line. "Hey Claude, our stuff needs to make more money. We are at risk for losing more." "Rest assured, the 'situation' will only worsen if you resist our benevolent offer." "We're aren't even at AGI yet, but I for one welcome our new agentic overlords."

toasty228 3mo ago

That's what we've been doing with tech for 200+ years, why would it stop now. Build cool shit now and let future generations handle the problems

pton_xd 3mo ago

That begs the question, what's the plan if AI does not help us understand the situation?

xlayn 3mo ago

This should be the top comment, the implication is: AI will decide if AI is correct or no, so adding a new layer of approval... Whenever you say, hey that's incorrect, the legal process goes to AI and will decide on that issue... Will work as amazing as the chatbots of the AI companies to solve your issues...

andsoitis 2mo ago
Also, Hope is not a strategy.