6 ms·

More like: "Let me clear a huge misunderstanding here by using poorly-defined terms and cramming niche complex ideas elaborated elsewhere into this tweet." Som

More like: "Let me clear a huge misunderstanding here by using poorly-defined terms and cramming niche complex ideas elaborated elsewhere into this tweet." Someone correct me, but it seems like he's saying: 1- generative models don't understand the real world 2- generative models that work off of just pixels are more expensive and less useful than a model that represents the contents of the frame with abstract representations, particularly when related to higher-level actions like applying logic to a scene and not just interpolating between frames. 3- V-JEPA is such a model, and it performs particuarly well against generative when combining those representations as inputs with a much more easily trained and small model trained on specific tasks.

vasco 3y ago
I'm nowhere near an expert but it seemed like he was claiming true understanding of latent space is needed for generating coherent continuations, but Sora demo already has longish videos that are coherent. It's hard for me not to think this is someone just trying to still be right when they are wrong, but I may misunderstand.
- justanotherjoe 3y ago
  The way i understand it is that Sora is mostly just 'moving pictures' with no rhyme or reason. Yann Lecun is interested in videos that tell a 'story', with cause and effect. Like a magician putting his hand into a top hat and pulling out a rabbit, kind of video.
  - throw310822 3y ago
    Yeah, the same way ChatGPT is only predicting the next word with no rhyme or reason. However to actually predict the next word so that the entire sentence makes sense and is relevant for the context (e.g. answers a question) you probably must be actually understanding the meaning of the words and the language and have a world model. I can't imagine a NN moving around pixels in the shape of a cat with no understanding whatsoever of what a cat is, what it can do, how it is supposed to move, what it wants, etc.
    - justanotherjoe 3y ago
      Of course. But stable diffusion can do that. It understands what a cat is and can draw cat wearing a hat. Videos are about actions, cause and effect, which is entirely different thing than still pictures.
      - throw310822 3y ago
        Well, we definitely see actions, and actors, and causes and effects in the SORA demo videos. With imperfections and occasional mistakes, but they're undeniably there.
      - dkjaudyeqooe 3y ago
        It doesn't understand a cat at all. Humans understand, models are deterministic functions with some randomness added in. Just because it appears to understand doesn't make it so.
        deleted 3y ago
        [deleted]
        justanotherjoe 3y ago
        diffusion models can reliably draw a cat when prompted a cat and given a random noise. Sure it's deterministic, but it can work with any random noise in a explorative kind of way. I'd say it's very general in it's 'understanding' of a cat.
        TheOtherHobbes 3y ago
        It has no "understanding" of a cat. It's an associative store with soft edges that pulls out compressed cat representations when given the noun "cat". The key store includes nouns, adverbs, verbs, and adjectives, and style abstractions, and there are mappings into the store that link all of those. But they're very limited, and if you prompt with a relationship that isn't defined you get best-guess, which will either be quite distant or contaminated with other values. If you ask Dall-E for "a woman made of birds" you get a composite that also includes trees and/or leaves. Dall-E has values for "made of" and "birds" but its value representation for "birds" is contaminated with contextual trees and branches. Leonardo doesn't have a value for "made of", so you get a woman surrounded by bird-like blobs. To understand a cat in a human sense the store would have to include the shape, the movement dynamics in all possible situations, the textures, and a set of defining behaviours that is as complete as possible. It would also have to be able to provide and remember an object-constant instantiation of a specific cat that is clean of contamination. SORA is maybe 10% of the way there. One of the examples doing the rounds shows some puppies playing in snow. It looks impressive until you realise the puppies are zombies. They have none of the expressions or emotions of a real puppy. None of this is impossible, but training time, storage, and power consumption all explode the more information you try to include.
  - vasco 3y ago
    Did you see the bling zoo demo? How can you say it doesn't have rhyme or reason?
  - hackerlight 3y ago
    I want him to answer how Sora can do water simulations without having a model of part of physics. How can Sora predict where the waves and ripples should go? Is it just "correlations not causal", whatever that means.
    - gtirloni 3y ago
      Generative AI is already full of misunderstandings. From people claiming it "understands" to now that it "simulates". I'm no math wiz and my training in statistics is severely lacking but it feels like people need to review what they think it's possible with Generative AI because we are so far from understanding and AGI that my head hurts every time these words show up in a discussion.
      - hackerlight 3y ago
        Geoffrey Hinton and Yoshua Bengio says it can understand and we are close to AGI. Maybe you can explain why they're wrong instead of just saying it.
        gtirloni 3y ago
        Extraordinary claims require extraordinary proof. I'm not the one making the claims.
aaron695 3y ago
Video, he says 1. The next frame is easy, but multiple frames is not 2. What works for text doesn't work for video. Then Sora comes out and shows multiple frames and someone tweets gotcha. He then tweets without saying he misspoke ..... goes on about the model doesn't understand physics. And his project, V-JEPA, is the best He keeps saying stuff about "sucks as a mental model" but doesn't say why that would not apply to text. https://twitter.com/ylecun/with_replies https://twitter.com/ylecun/with_replies Me: If text doesn't need a mental model, I see no reason video needs it. His argument sucks or is badly worded.

vasco 3y ago

I'm nowhere near an expert but it seemed like he was claiming true understanding of latent space is needed for generating coherent continuations, but Sora demo already has longish videos that are coherent. It's hard for me not to think this is someone just trying to still be right when they are wrong, but I may misunderstand.

justanotherjoe 3y ago
The way i understand it is that Sora is mostly just 'moving pictures' with no rhyme or reason. Yann Lecun is interested in videos that tell a 'story', with cause and effect. Like a magician putting his hand into a top hat and pulling out a rabbit, kind of video.
- throw310822 3y ago
  Yeah, the same way ChatGPT is only predicting the next word with no rhyme or reason. However to actually predict the next word so that the entire sentence makes sense and is relevant for the context (e.g. answers a question) you probably must be actually understanding the meaning of the words and the language and have a world model. I can't imagine a NN moving around pixels in the shape of a cat with no understanding whatsoever of what a cat is, what it can do, how it is supposed to move, what it wants, etc.
  - justanotherjoe 3y ago
    Of course. But stable diffusion can do that. It understands what a cat is and can draw cat wearing a hat. Videos are about actions, cause and effect, which is entirely different thing than still pictures.
    - throw310822 3y ago
      Well, we definitely see actions, and actors, and causes and effects in the SORA demo videos. With imperfections and occasional mistakes, but they're undeniably there.
    - dkjaudyeqooe 3y ago
      It doesn't understand a cat at all. Humans understand, models are deterministic functions with some randomness added in. Just because it appears to understand doesn't make it so.
      - deleted 3y ago
        [deleted]
      - justanotherjoe 3y ago
        diffusion models can reliably draw a cat when prompted a cat and given a random noise. Sure it's deterministic, but it can work with any random noise in a explorative kind of way. I'd say it's very general in it's 'understanding' of a cat.
        TheOtherHobbes 3y ago
        It has no "understanding" of a cat. It's an associative store with soft edges that pulls out compressed cat representations when given the noun "cat". The key store includes nouns, adverbs, verbs, and adjectives, and style abstractions, and there are mappings into the store that link all of those. But they're very limited, and if you prompt with a relationship that isn't defined you get best-guess, which will either be quite distant or contaminated with other values. If you ask Dall-E for "a woman made of birds" you get a composite that also includes trees and/or leaves. Dall-E has values for "made of" and "birds" but its value representation for "birds" is contaminated with contextual trees and branches. Leonardo doesn't have a value for "made of", so you get a woman surrounded by bird-like blobs. To understand a cat in a human sense the store would have to include the shape, the movement dynamics in all possible situations, the textures, and a set of defining behaviours that is as complete as possible. It would also have to be able to provide and remember an object-constant instantiation of a specific cat that is clean of contamination. SORA is maybe 10% of the way there. One of the examples doing the rounds shows some puppies playing in snow. It looks impressive until you realise the puppies are zombies. They have none of the expressions or emotions of a real puppy. None of this is impossible, but training time, storage, and power consumption all explode the more information you try to include.
- vasco 3y ago
  Did you see the bling zoo demo? How can you say it doesn't have rhyme or reason?
- hackerlight 3y ago
  I want him to answer how Sora can do water simulations without having a model of part of physics. How can Sora predict where the waves and ripples should go? Is it just "correlations not causal", whatever that means.
  - gtirloni 3y ago
    Generative AI is already full of misunderstandings. From people claiming it "understands" to now that it "simulates". I'm no math wiz and my training in statistics is severely lacking but it feels like people need to review what they think it's possible with Generative AI because we are so far from understanding and AGI that my head hurts every time these words show up in a discussion.
    - hackerlight 3y ago
      Geoffrey Hinton and Yoshua Bengio says it can understand and we are close to AGI. Maybe you can explain why they're wrong instead of just saying it.
      - gtirloni 3y ago
        Extraordinary claims require extraordinary proof. I'm not the one making the claims.

aaron695 3y ago

Video, he says 1. The next frame is easy, but multiple frames is not 2. What works for text doesn't work for video. Then Sora comes out and shows multiple frames and someone tweets gotcha. He then tweets without saying he misspoke ..... goes on about the model doesn't understand physics. And his project, V-JEPA, is the best He keeps saying stuff about "sucks as a mental model" but doesn't say why that would not apply to text. https://twitter.com/ylecun/with_replies https://twitter.com/ylecun/with_replies Me: If text doesn't need a mental model, I see no reason video needs it. His argument sucks or is badly worded.