3 ms·

The current AIs are pretty bad at handling negation especially when the models are small and quantised. To be fair, so are humans: double, triple, or even highe

The current AIs are pretty bad at handling negation especially when the models are small and quantised. To be fair, so are humans: double, triple, or even higher negatives can trip people up. This effect of smaller models being bad at negation is most obvious in image generators, most of which are only a handful of gigabytes in size. If you ask one for “don’t show an elephant next to the circus tent!” then you will definitely get an elephant.

echoangle 2y ago
Isn’t the negative prompting thing with image generators just how they work? As far as I understand, the problem is that training data isn’t normally annotated with „no elephant“ with all images without elephant, so putting „no elephant“ in the prompt most closely matches training data that’s annotated with „elephant“ and includes elephants. The image models aren’t really made to understand proper sentences, I think.
- jiggawatts 2y ago
  Yes, but it’s more complex than that! If you ask “who is Tom Cruise’s mother” you will get a much more robust response than asking “who is Mary Lee Pfeiffer’s son?”. It’s not just negation that models struggle with, but also reversing the direction of any arrow connecting facts, or wandering too far from established patterns of any kind. It’s been studied scientifically and is one of most fascinating aspects because it also reveals the weaknesses and flaws of human thinking. Researchers are already trying to fix this problem by generating synthetic training data that includes negations and reversals. That makes you wonder: would this approach improve the robustness of human education also?
  - whycome 2y ago
    This is a super interesting line of info. Thank you! I didn't think of it as a negation-specific challenge but that's really cool insight. "Don't think of an elephant." It's actually interesting how often we have to guess that someone dropped a "not" in conversation based on the context. It wouldn't be hard to have an iMessage bot (eg on a Mac) running to test some of this out on the fly.

echoangle 2y ago

Isn’t the negative prompting thing with image generators just how they work? As far as I understand, the problem is that training data isn’t normally annotated with „no elephant“ with all images without elephant, so putting „no elephant“ in the prompt most closely matches training data that’s annotated with „elephant“ and includes elephants. The image models aren’t really made to understand proper sentences, I think.

jiggawatts 2y ago
Yes, but it’s more complex than that! If you ask “who is Tom Cruise’s mother” you will get a much more robust response than asking “who is Mary Lee Pfeiffer’s son?”. It’s not just negation that models struggle with, but also reversing the direction of any arrow connecting facts, or wandering too far from established patterns of any kind. It’s been studied scientifically and is one of most fascinating aspects because it also reveals the weaknesses and flaws of human thinking. Researchers are already trying to fix this problem by generating synthetic training data that includes negations and reversals. That makes you wonder: would this approach improve the robustness of human education also?
- whycome 2y ago
  This is a super interesting line of info. Thank you! I didn't think of it as a negation-specific challenge but that's really cool insight. "Don't think of an elephant." It's actually interesting how often we have to guess that someone dropped a "not" in conversation based on the context. It wouldn't be hard to have an iMessage bot (eg on a Mac) running to test some of this out on the fly.