3 ms·

You provide no actual arguments as to why LLMs are fundamentally unable to learn this. Your doubt is as valuable as my confidence.

AIPedant 2y ago
Well, it's a good thing I didn't say "fundamentally unable to learn this"! I said that learning visual reasoning from video is probably not enough: if you claim it is enough, you have to reconcile that with failures in Sora, Veo 2, etc. Veo 2's problems are especially serious since it was trained on an all-DeepMind-can-eat diet of YouTube videos. It seems like they need a stronger algorithm, not more Red Dead Redemption 2 footage.
- dinfinity 2y ago
  > I said that learning visual reasoning from video is probably not enough Fair enough; you did indeed say that. > if you claim it is enough, you have to reconcile that with failures in Sora, Veo 2, etc. This is flawed reasoning, though. The current state of video generating AI and the completeness of the training set does not reliably prove that the network used to perform the generation is incapable of physical modeling and/or object permanence. Those things are ultimately (the modeling of) relations between past and present tokens, so the transformer architecture does fit. It might just be a matter of compute/network size (modeling four dimensional physical relations in high resolution is pretty hard, yo). If you look at the scaling results from the early Sora blogs, the natural increase of physical accuracy with more compute is visible: https://openai.com/index/video-generation-models-as-world-simulators/#scaling-transformers-for-video-generation https://openai.com/index/video-generation-models-as-world-si... It also might be a matter of fine-tuning training on (and optimizing for) four dimensional/physical accuracy rather than on "does this generated frame look like the actual frame?"
  - zveyaeyv3sfye 2y ago
    [flagged]
viccis 2y ago
Because the nature of their operation (learning a probability distribution over a corpus of observed data) is not the same as creating synthetic a priori knowledge (object permanence is a case of cause and effect which is synthetic a priori knowledge). All LLM knowledge is by definition a posteriori.
- AstralStorm 2y ago
  That LLMs cannot synthesize it into a propri, including other rules of logic and mathematics, is a major failure of the technology...

AIPedant 2y ago

Well, it's a good thing I didn't say "fundamentally unable to learn this"! I said that learning visual reasoning from video is probably not enough: if you claim it is enough, you have to reconcile that with failures in Sora, Veo 2, etc. Veo 2's problems are especially serious since it was trained on an all-DeepMind-can-eat diet of YouTube videos. It seems like they need a stronger algorithm, not more Red Dead Redemption 2 footage.

dinfinity 2y ago
> I said that learning visual reasoning from video is probably not enough Fair enough; you did indeed say that. > if you claim it is enough, you have to reconcile that with failures in Sora, Veo 2, etc. This is flawed reasoning, though. The current state of video generating AI and the completeness of the training set does not reliably prove that the network used to perform the generation is incapable of physical modeling and/or object permanence. Those things are ultimately (the modeling of) relations between past and present tokens, so the transformer architecture does fit. It might just be a matter of compute/network size (modeling four dimensional physical relations in high resolution is pretty hard, yo). If you look at the scaling results from the early Sora blogs, the natural increase of physical accuracy with more compute is visible: https://openai.com/index/video-generation-models-as-world-simulators/#scaling-transformers-for-video-generation https://openai.com/index/video-generation-models-as-world-si... It also might be a matter of fine-tuning training on (and optimizing for) four dimensional/physical accuracy rather than on "does this generated frame look like the actual frame?"
- zveyaeyv3sfye 2y ago
  [flagged]

viccis 2y ago

Because the nature of their operation (learning a probability distribution over a corpus of observed data) is not the same as creating synthetic a priori knowledge (object permanence is a case of cause and effect which is synthetic a priori knowledge). All LLM knowledge is by definition a posteriori.

AstralStorm 2y ago
That LLMs cannot synthesize it into a propri, including other rules of logic and mathematics, is a major failure of the technology...