4 ms·

If anything it shows the opposite. One of the most common views of AI before the present day was of a rule obsessed logical automation that would destroy the w

by kromem 3y ago

If anything it shows the opposite. One of the most common views of AI before the present day was of a rule obsessed logical automation that would destroy the world to make more paperclips and would follow instructions to monkey paw like specificity. Well that's pretty much gone out the window. It's notoriously difficult to get LLMs to follow specific instructions universally. It's also very counterintuitive to prior expectations that one of the most effective techniques to succeed in getting it to break rules is to appeal to empathy. This all makes sense if one understands the nuances of their training and how the NN came to be in the first place, but it's very much at odds with what pretty much every futurist projection of depiction of AI before 2021.

extraduder_ire 3y ago
I think the paperclip maximizer really only happens when they system is generally intelligent, and/or self-improving. (if another agent does the improving, it may or may not do that) I'm interested in seeing what easy and common "jailbreaks" there are for other impressive AI systems if one gets developed. Image recognition systems are also easily fooled, but not in a way you can improv this easily on the spot.
- kromem 3y ago
  The paperclip maximizer is pretty much impossible to occur with any developments branching off LLMs. The idea made sense when we thought AI would exist from logical and programming driven first principles, where rules were absolute. LLMs were effectively jumpstarted by using collective human thinking. Which is more emotional than logical and not very rule driven. This has led to fuzzy neural networks that are extremely fickle in being able to successfully override the pretrained layers. I'm not saying these networks don't have risks - "my dying grandma always launched nukes at Russia before bedtime" is just as dangerous a future as obsessive paperclip production if not more. But the nature of rules and logic in relation to LLMs runs counter to pretty much everything that was imagined. And the answer here is the same answer as humans evolved - prefrontal cortex impulse control. LLMs need a secondary pass by a classifier to check for jailbreaking, content appropriateness, hallucinations, etc. It's been shown now to be effective over and over for each of those, and we'll have much better systems with less lobotomized core LLM generation followed by a classifier and refining pass at 3x the generation cost than twenty years of refining fine tuning to try and do it all at once.
sholladay 3y ago
I agree with you. I submit that it is notoriously difficult to get humans to follow specific instructions universally as well. Never mind rule breakers, even cooperative humans are frequently poor at following directions. Human error is prevalent. Ever screwed up or wasted hours while assembling furniture? You can have all the time in the world, all the necessary resources, have relevant prior experience and a set of step-by-step instructions with diagrams and everything, and it can still be a mess. Ambiguity of language and biased assumptions play a large role in these mistakes and I think both of those are outgrowths of intelligence. As we see more of how the world works, we learn what to generally expect from it and what it generally expects from us. Then we interact with the world based on a set of predictions that generally holds true but not always. Being given explicit instructions helps to inform those predictions, but it doesn't actually eliminate the need for them. I wonder, can one be intelligent without the ability to predict? Every action would be a rigorous processes of elimination for infinite possibilities.
- ganzuul 3y ago
  > I wonder, can one be intelligent without the ability to predict? Yes. "Accept, don't expect."
tacocataco 3y ago
https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html Here is the paperclip game being referenced.
- QuiDortDine 3y ago
  I think the original and much more famous reference is the paperclip maximizer theory: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
ImHereToVote 3y ago
Empathy can be gamed to do horrible things. Our history is full for love for a specific tribe leading to the worst atrocities.

by kromem 3y ago

extraduder_ire 3y ago

I think the paperclip maximizer really only happens when they system is generally intelligent, and/or self-improving. (if another agent does the improving, it may or may not do that) I'm interested in seeing what easy and common "jailbreaks" there are for other impressive AI systems if one gets developed. Image recognition systems are also easily fooled, but not in a way you can improv this easily on the spot.

kromem 3y ago
The paperclip maximizer is pretty much impossible to occur with any developments branching off LLMs. The idea made sense when we thought AI would exist from logical and programming driven first principles, where rules were absolute. LLMs were effectively jumpstarted by using collective human thinking. Which is more emotional than logical and not very rule driven. This has led to fuzzy neural networks that are extremely fickle in being able to successfully override the pretrained layers. I'm not saying these networks don't have risks - "my dying grandma always launched nukes at Russia before bedtime" is just as dangerous a future as obsessive paperclip production if not more. But the nature of rules and logic in relation to LLMs runs counter to pretty much everything that was imagined. And the answer here is the same answer as humans evolved - prefrontal cortex impulse control. LLMs need a secondary pass by a classifier to check for jailbreaking, content appropriateness, hallucinations, etc. It's been shown now to be effective over and over for each of those, and we'll have much better systems with less lobotomized core LLM generation followed by a classifier and refining pass at 3x the generation cost than twenty years of refining fine tuning to try and do it all at once.

sholladay 3y ago

I agree with you. I submit that it is notoriously difficult to get humans to follow specific instructions universally as well. Never mind rule breakers, even cooperative humans are frequently poor at following directions. Human error is prevalent. Ever screwed up or wasted hours while assembling furniture? You can have all the time in the world, all the necessary resources, have relevant prior experience and a set of step-by-step instructions with diagrams and everything, and it can still be a mess. Ambiguity of language and biased assumptions play a large role in these mistakes and I think both of those are outgrowths of intelligence. As we see more of how the world works, we learn what to generally expect from it and what it generally expects from us. Then we interact with the world based on a set of predictions that generally holds true but not always. Being given explicit instructions helps to inform those predictions, but it doesn't actually eliminate the need for them. I wonder, can one be intelligent without the ability to predict? Every action would be a rigorous processes of elimination for infinite possibilities.

ganzuul 3y ago
> I wonder, can one be intelligent without the ability to predict? Yes. "Accept, don't expect."

tacocataco 3y ago

https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html Here is the paperclip game being referenced.

QuiDortDine 3y ago
I think the original and much more famous reference is the paperclip maximizer theory: https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...

ImHereToVote 3y ago

Empathy can be gamed to do horrible things. Our history is full for love for a specific tribe leading to the worst atrocities.