2 ms·

I think the paperclip maximizer really only happens when they system is generally intelligent, and/or self-improving. (if another agent does the improving, it m

• kromem 3y ago
  The paperclip maximizer is pretty much impossible to occur with any developments branching off LLMs. The idea made sense when we thought AI would exist from logical and programming driven first principles, where rules were absolute. LLMs were effectively jumpstarted by using collective human thinking. Which is more emotional than logical and not very rule driven. This has led to fuzzy neural networks that are extremely fickle in being able to successfully override the pretrained layers. I'm not saying these networks don't have risks - "my dying grandma always launched nukes at Russia before bedtime" is just as dangerous a future as obsessive paperclip production if not more. But the nature of rules and logic in relation to LLMs runs counter to pretty much everything that was imagined. And the answer here is the same answer as humans evolved - prefrontal cortex impulse control. LLMs need a secondary pass by a classifier to check for jailbreaking, content appropriateness, hallucinations, etc. It's been shown now to be effective over and over for each of those, and we'll have much better systems with less lobotomized core LLM generation followed by a classifier and refining pass at 3x the generation cost than twenty years of refining fine tuning to try and do it all at once.