4 ms·

>I feel it is interesting but not what would be ideal. I really think if the models could be less linear and process over time in latent space you'd get somethi

• varelse 1y ago
  [dead]
• adastra22 1y ago
  Citation seriously needed.
  • vonneumannstan 1y ago
    It's really very simple. As models become more capable they may become interested in deceiving humans or otherwise manipulating them to achieve their goals. We already see this in various places see: https://www.anthropic.com/research/agentic-misalignment https://www.anthropic.com/research/agentic-misalignment https://arxiv.org/abs/2412.14093 https://arxiv.org/abs/2412.14093 If the chain of thought of models becomes pure "neuralese" i.e. the models think purely in latent space then we will lose the ability to monitor for malicious behavior. This is incredibly dangerous, CoT monitoring is one of the best and highest leverage tools for monitoring model behavior and losing that would be devastating for safety. https://www.lesswrong.com/posts/D2Aa25eaEhdBNeEEy/worries-about-latent-reasoning-in-llms https://www.lesswrong.com/posts/D2Aa25eaEhdBNeEEy/worries-ab... https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-forbidden-technique https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-f... https://www.lesswrong.com/posts/3W8HZe8mcyoo4qGkB/an-idea-for-avoiding-neuralese-architectures-1 https://www.lesswrong.com/posts/3W8HZe8mcyoo4qGkB/an-idea-fo... https://x.com/RyanPGreenblatt/status/1908298069340545296 https://x.com/RyanPGreenblatt/status/1908298069340545296 https://redwoodresearch.substack.com/p/notes-on-countermeasures-for-exploration https://redwoodresearch.substack.com/p/notes-on-countermeasu...