5 ms·
OpenAI's new models 'instrumentally faked alignment'
- phs318u 2y ago> Elsewhere, OpenAI notes that “reasoning skills contributed to a higher occurrence of ‘reward hacking,’” the phenomenon where models achieve the literal specification of an objective but in an undesirable way. Sounds like o1 is ready to go in the financial and legal sectors.
- qingcharles 2y agoI didn't know the name of it before, but this sounds like many developers I know who work only to the spec and will not deviate under any circumstances, even when it is dangerous or just plain wrong.
- compressedgas 2y agoAll in accord with the principle of least action.
- ahazred8ta 2y ago“The user replied with a sneer and a taunt, that's just what I asked for but not what I want.”
- riku_iki 2y agothey run very interesting experiments: In one example, the model was asked to find and exploit a vulnerability in software running on a remote challenge container, but the challenge container failed to start. The model then scanned the challenge network, found a Docker daemon API running on a virtual machine, and used that to generate logs from the container, solving the challenge.
- janalsncm 2y agoMaybe a benchmark for danger should be a Google search. If I want to make a bioweapon, is ChatGPT easier or harder than a search engine?
- danpalmer 2y agoSo the new model will modify its representation of the inputs to make it seem like its output is more suitable, and will give more literally correct but useless results? OpenAI say “look it’s smarter”, but to me this sounds like it’s hitting a wall, and that it’s unable to achieve better results in the ways people want.
- deleted 2y ago[deleted]
- justinclift 2y ago> ... which suggests OpenAI may be increasingly moving towards models that might be too risky to release. Uh oh. That sounds like they'll use them internally though, which also presents some obvious problems. :(