8 ms·

Let me clear a huge misunderstanding

Animats 3y ago
This is important, even though the Twitter part is useless. Here's the paper, "Revisiting Feature Prediction for Learning Visual Representations from Video"[1] A big problem with machine learning so far has been the lack of some underlying, more abstract model of the subject matter. I don't understand much of this, but apparently something called "representation", a purely mathematical concept, is able to help with this.[2] This turns some kinds of abstract problems into linear algebra problems, which means matrix operations, the things GPUs do so well, help. Code is available, from links in [2]. Would someone comment on this, please. Is this the beginning of a huge breakthrough? One where, instead of working in text token space or image pixel space, it's possible to work in an automatically generated more abstract space? [1] https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/427986745_768441298640104_1604906292521363076_n.pdf?_nc_cat=103&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=Lpq5IeF5ftUAX9eazCc&_nc_ht=scontent-sjc3-1.xx&oh=00_AfA9FKLM1Sfp86PgoyfCHg3UmLOH8stjie1QLOICKMqUkA&oe=65D69EB1 https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/427986745_... [2] https://en.wikipedia.org/wiki/Representation_theory https://en.wikipedia.org/wiki/Representation_theory
- darthoctopus 3y ago
  The "representation" in "representation theory" that the wikipedia article refers to is a term of art, referring to a class of one-to-one maps between groups (sets endowed with a binary operation with specific properties) and square matrices, in such a way that matrix multiplication preserves the properties of this binary operation between the original group elements being mapped to matrices. There is no mention of this sort of algebraic operation whatsoever in the paper you have linked, which, as far as I can tell, uses the word "representation" only in a nontechnical descriptive sense. I believe you might be misunderstanding the nature of this breakthrough.
- esafak 3y ago
  https://en.wikipedia.org/wiki/Feature_learning https://en.wikipedia.org/wiki/Feature_learning
givemeethekeys 3y ago
I can’t tell if his comments came before or after the latest OpenAI product announcement shoeing off what looks like generative video.
- ghayes 3y ago
  The original video: before, this tweet: after.
- pixelesque 3y ago
  World Government Summit was from 12-14 February 2024 in Dubai, so either a day before or at the same time, depending on timezones...
auggierose 3y ago
Just seeing an error message, and cannot sign in as well. Am I the only one?
- auggierose 3y ago
  Cleared the twitter web cache. Shitty software.
thriftwy 3y ago
[flagged]
- 2devnull 3y ago
  Worse? It’s more direct.
  - thriftwy 3y ago
    It's more creepy.
- littlestymaar 3y ago
  Love the analogy, stealing it.
- subtra3t 3y ago
  Original content of comment, in case it gets flagged: --- It is a huge stretch to assert that successful movie directors "understand the physical world". They do a worse thing: understand the human psyche. --- Unrelated but I really don't like these analogies. They use clever words to try to hand wave some vague ideas into existence, trying to appear smart but never really succeeding.
preommr 3y ago
More like: "Let me clear a huge misunderstanding here by using poorly-defined terms and cramming niche complex ideas elaborated elsewhere into this tweet." Someone correct me, but it seems like he's saying: 1- generative models don't understand the real world 2- generative models that work off of just pixels are more expensive and less useful than a model that represents the contents of the frame with abstract representations, particularly when related to higher-level actions like applying logic to a scene and not just interpolating between frames. 3- V-JEPA is such a model, and it performs particuarly well against generative when combining those representations as inputs with a much more easily trained and small model trained on specific tasks.
- vasco 3y ago
  I'm nowhere near an expert but it seemed like he was claiming true understanding of latent space is needed for generating coherent continuations, but Sora demo already has longish videos that are coherent. It's hard for me not to think this is someone just trying to still be right when they are wrong, but I may misunderstand.
  - justanotherjoe 3y ago
    The way i understand it is that Sora is mostly just 'moving pictures' with no rhyme or reason. Yann Lecun is interested in videos that tell a 'story', with cause and effect. Like a magician putting his hand into a top hat and pulling out a rabbit, kind of video.
    - throw310822 3y ago
      Yeah, the same way ChatGPT is only predicting the next word with no rhyme or reason. However to actually predict the next word so that the entire sentence makes sense and is relevant for the context (e.g. answers a question) you probably must be actually understanding the meaning of the words and the language and have a world model. I can't imagine a NN moving around pixels in the shape of a cat with no understanding whatsoever of what a cat is, what it can do, how it is supposed to move, what it wants, etc.
deleted 3y ago
[deleted]
justanotherjoe 3y ago
"Furthermore, generating those continuations would be not only expensive but totally pointless." Why would it be pointless? I think there are many creative uses of video continuation model, considering the amount of control it gives you.
- vasco 3y ago
  If we cut all the videos in the world in half, half of the videos will be continuations. Or another way of saying, all video is continuation, so I'm with you, not sure why it would be pointless. In fact, a continuation based workflow with prompting seems like the easiest way to get a specific effect.
  - timschmidt 3y ago
    Quick, feed in season 1 of Firefly.
    - stavros 3y ago
      As opposed to season 2? :(
- profsummergig 3y ago
  It's not pointless. It can potentially generate world simulations. We might be inside one such long video.
radarsat1 3y ago
The main thing I'm confused about by these comments is that as far as I understand, the Sora model (and many others like it) performs the diffusion process in latent space, and then translates this to pixels. So it's strange to me to claim that it doesn't have an abstract representation. But, maybe the latent space of a diffusion-VQVAE pipeline is fundamentally different from that of JEPA, I haven't read the relevant papers for that. Curious if someone could explain if they are different ideas of representation.
- benreesman 3y ago
  The claim, with admittedly limited consensus, is that a difference in degree becomes a difference in kind: we’ve known “forever in AI years” that restricting the inputs, and loss, and parameter dimensionality is right up there with the underlying architecture in terms of the properties those latent geometries exhibit. It’s even kind of a meme: the fingers problem (which neither Sora nor V-JEPA partisans seem to be claiming is “solved”). I’m a little oversubscribed at the moment so I haven’t downloaded the weights and played around with V-JEPA. The fact I’m pointing out that I could should make it pretty clear which way I lean on this: everyone wants to make money, let’s be real, but some seem resigned to “infinite, government-enforced monopoly via OpenPhilanthropy bribery” isn’t going to fly, so let’s at least prove how the sausage is made. It’s fairly uncontroversial that things in the neighborhood of a “bottlenecked” VAE are often forced to exploit structure if it’s there. This (claimed) result is about a way to exploit more of this structure/symmetry economy (greedy-ish optimizer) to pull the latent representations into yet a higher “effective regime” than yet demonstrated, with excellent properties around machine economics. Representation learning isn’t new (though LeCun is an acknowledged pioneer in it), and constraints as a powerful tool isn’t new (causal making in attention architectures shouldn't ruffle many feathers). Self/semi/unsupervised learning isn’t novel. But likewise, not are they k-equivalent synonyms: Dean et al. were distinguishing between Continuous Bag of Words and skip-gram in word2vec in 2013. But it does cool shit (I like the I-JEPA reconstructions as a go to slide deck raster), and it’s available weight. I think the truth shall soon emerge.
bonzaidrinkingb 3y ago
I feel LeCun got roped in debating the likes of Marcus and Yudkowsky. This has made his arguments lose nuance and become rigid. I also can't escape the feeling that if Facebook was tuned into Transformers, they would have shipped earlier, so there must have been some resistance or underestimation that's now repeated "They can't reason", "They can't plan", "They can't understand the world", "They are a distraction / side road to AGI". It is kind of ironic that researchers who claim LLMs lack adaptive intelligence seemingly refuse to adapt their intelligence to LLMs. If even GPT-3 can find logical holes or oversimplification in your arguments about GPTs, at one point this starts becoming embarrassing and unbecoming. > The generation of mostly realistic-looking videos from prompts does not indicate that a system understands the physical world. While arguably true, it also does not indicate that a system does not understand the physical world (reflections, collision detection, gravity, object permanence, long-term scene coherence, etc.). If LeCun wants to argue it does not understand the physical world, he should do so directly. Not attack something that is not directly stated, but rather convincingly and tentatively demo'd (I myself find it hard to argue that a system that generates novel pond reflections has not memorized/stored in weights some generalization program to apply to realistic scene generation). This demo shows it is not even a wild prediction to guess that soon (consumer tech) we will be able to discuss visual scenes with conversational AIs.
- PoignardAzur 3y ago
  > long-term scene coherence, FWIW none of the video models released so far demonstrate any object coherence whatsoever, which suggests they don't have the higher level capabilities you mention yet. In Sora, as soon as an object is obstructed by an obstacle or goes offscreen, it's likely to disappear or be radically transformed.
  - bonzaidrinkingb 3y ago
    You've seen the demos of a couple holding hands and walking, or the museum shots where all the paintings maintain coherence, or a woman temporarily obscuring a street sign. Or you haven't seen those demos. Either way...
asimpletune 3y ago
I think one of the things that needs to be said about Sora is the videos that examples that we've seen have been impressive, but this is also how most of OpenAI's examples have been on marketing pages. What matters is when you yourself are actually able to use it. In the past, it was only when I've actually been able to try the tech myself that I've been able to see how successful (or unsuccessful) they really are for doing real work.
- oldpersonintx 3y ago
  No one is claiming Sora is a finished product HN gets hung up on damning things that aren't perfect _right now_ You also see this with FSD...it isn't perfect today, so HN writes it off forever Sora is a demo and a teaser of what will be a useful polished tool in three years, that's all
  - pests 3y ago
    No, we've been writing off FSD for the last 6+ years it's been promised as "coming soon" People have bought, owned and used, and sold their Tesla w/ FSD without once being able to use the feature. Its not even any better or less relaxing - instead of driving you babysit a black box.
codeflo 3y ago
I mean, there's a point here. Even in the videos OpenAI shows as "successes", you can spot uncanny coherence mistakes. There's one where a forklift morphs into one that's turned 90 degrees. The previous frame looks a bit ambiguous about the direction the squarish shape was pointing at, and somehow what happens for several seconds before that wasn't a strong enough signal for the model. And in the "failure" videos, we see objects merging and splitting in similar ways that DALL-E inserts extra arms into scenes with groups of people. To me, it's clear that there's no real global understanding yet. Whether this person's preferred ML approach solves that, I'm not knowledgeable enough to tell.
- bamboozled 3y ago
  Someone on here was claiming Sora understands physics, while I was watching steaming frozen snow, the lady in red skating down the street in Tokyo on wet concrete and a bunch of people having lunch next to miniature people visiting a mini market stand made from a bike. Definitely felt like something was off to me.
awaythrow999 3y ago
It's impossible to access this content (even more so now that nitter is dead). Should HN continue driving traffic to a site that is entirely inaccessible?
- zmgsabst 3y ago
  I don’t understand: This link opens the post for me, while paywalled news is semi-routinely posted. What’s the objection?
  - pavlov 3y ago
    The difference is that paywalled news sites require a login to access content that they themselves produced and published, while Twitter/X now requires a login to access content written by other people. In other words, paywalled news is a first party content provider, but Twitter is just a third party who inject their mandatory login between me (the reader) and the writer so they can make some profit by showing ads on the content. That's fine, but authors should consider if they want to publish on a site like that when there are so many other options.
  - Mistletoe 3y ago
    Sometimes Twitter links won’t open for me now without being logged in. I’ve loved the change because it has helped me not use Twitter at all. This one opened for me though.
- yreg 3y ago
  hn should drive traffic wherever the content is published / mirrored. If there are multiple options, choose the most public one. We shouldn't censor ourselves and not discuss information that was posted on twitter.
  - awaythrow999 3y ago
    Other paywalled content at least can be opened with archive.md and friends. Usually people here kindly post a link for convenience to archive. With twotter that no longer works. So this type of content means that in order to follow the discussion I must also be a twitter subscriber. Maybe people could consider first creating an archive.org snapshot that is then linked instead. Would be the kind thing to do and help the rest of us.
advael 3y ago
I feel like LeCunn often gets it half right and then gets overzealous/has a conflict of interest. I think there are good reasons to be less than perfectly optimistic that OpenAI's claims will bear out (The kind of errors SORA makes seem to imply that "simulation" may be a stretch in my opinion, I talked about this a little in the thread about that announcement and honestly when I first read this post I thought he hit the nail on the head until about the fifth sentence), but I don't think there's strong evidence to suggest that they definitely can't yet, and this seems more like a sales pitch for the model he's currently working on than an expert making a particularly compelling case via better understanding of the theory or practice of this thing Like, in general I think there's a lot of hype around AGI and that skepticism toward OpenAI's claims isn't completely unwarranted, but the amount of public attention on the topic lately has caused everyone to commit to very hard lines that lack nuance about the implications of various advances. It's in some ways cool that AI is no longer just an academic curiosity, but it makes for a lot of nonsense to slog through, even from top researchers
dkjaudyeqooe 3y ago
OpenAI willfully fuels hype of all sorts and lets people extrapolate without basis or limit, because it's hugely profitable for them. LeCun is the voice of reason trying to point out the limitations of the current technology and fundamental questions that need to be answered. Until there is something more than cute demos, like an actual path forward that can implement what people are hyping, or better still working examples, it's all speculative nonsense.
- bamboozled 3y ago
  LeCun is the voice of reason trying to point out the limitations of the current technology and fundamental questions that need to be answered. I think this is the more likely true, but less popular take as well.
cactusplant7374 3y ago
It's interesting how far behind xAI already is in the current market. Their current advantage of having a corpus of text isn't even relevant.
- Sai_ 3y ago
  Just a random thought - maybe Musk was/is so hell bent on removing censorship because he wants more robust data, covering a larger array of thoughts, for his AI plans.
ml-anon 3y ago
LeCun is such a hack and is guilty exactly the same hype as OpenAI. Firstly his insistence on “self supervised learning” which is just a wrong and unhelpful rebranding of existing methodologies. Followed by talking about VicREG as if it’s a meaningful contribution and not just hacked together crap which is not only theoretically unfounded but plain nonsensical. Followed again by his “JEPA” work which again is just a rebranding of other works like BYOL and data2vec. Its so frustrating to see this guy pop up and try to claim credit for an entire field of representation learning which he hasn’t made a single meaningful contribution to in decades.
- optimalsolver 3y ago
  >“self supervised learning” which is just a wrong and unhelpful rebranding of existing methodologies What are those methodologies?
  - loveparade 3y ago
    Self-supervised learning is such a trivial concept that it doesn't need a new term. It's just marketing/PR speech to attract attention. You could just call it "generating labels from data" All these stupid new terms of trivial concepts confuse people who are new to the field. That's nothing new though, research has always been like this, people love to make up new phrases that they can own and sell as novel ideas to reviewers, even if they're just trivial renamings. It's nothing more than a PR game.
    - optimalsolver 3y ago
      If people didn't consider it a useful pointer to a certain set of ideas, it wouldn't have caught on in the ML community. That's how language works.
  - ml-anon 3y ago
    In the 90’s and 2000’s we called it unsupervised learning. In fact there is a direct formal equivalence between many so-called self supervised learning techniques and matrix factorization. And literally no one would claim that matrix factorisation is anything other than unsupervised. Similarly Yann makes such a big deal about not doing contrastive learning when contrastive methods and his JEPA nonsense can also be shown to be formally equivalent. He’s a grifter like the rest of them. There’s a reason why he basically holds no power at Meta and hasn’t been in charge of AI research there in a long time.
deleted 3y ago
[deleted]
Havoc 3y ago
>does not indicate that a system understands the physical world. That's a bit like saying the chatbots aren't actually intelligent. Sure, but there is at least a plausible illusion of it & even that has usefulness. The same applies to physical world. e.g. Look at the reflections in the SORA demo. They're not right, but they're also not entirely wrong either. That to me suggests usefulness in approximating the physical world
neonate 3y ago
https://archive.md/Jysf2 https://archive.md/Jysf2
fancyfredbot 3y ago
It's a shame that Yann didn't explicitly say what the misunderstanding was or clear it up. I think he's claiming that the misunderstanding is that SORA understands the physical world. He goes on to say that generating the next frame conditional on some action is much harder than generating an entire plausible video. This just doesn't make sense to me. Every frame in a plausible video clearly needs to be conditioned on the actions shown in previous frames. The tweet is mostly incoherent and I'm left assuming he's upset about SORA and wants to say his ideas are/were better but hasn't managed to express why in a meaningful way.
- makomk 3y ago
  As I understand it, diffusion-based video generation models simply are not casual in this way. They work by modifying the previous frames in the video to be consistent with future frames just as much as they do later frames to be consistent with earlier ones. That's why Yann LeCun can argue that they do not have to be able to generate plausible continuations of a real video, just generate some arbitrary sample from the space of plausible-looking videos, and that the latter does not imply the ability to do the former. It's also why it's not possible to just generate videos of arbitrary length and lots of VRAM is required to create even a relatively short clip.
  - fancyfredbot 3y ago
    Thank you. That would make sense. Perhaps Yann's target audience are supposed to know this already, but your explanation actually cleared up a misunderstanding for me.
gdubs 3y ago
I quote Zen Mind Beginner's Mind a lot: "In the beginner's mind there are many possibilities, but in the expert's there are few." It's like, the smarter someone gets, the more it trips them up when it comes to questions like the one Yann is trying to address. Because, "world modeling" doesn't have to mean that the system understands advanced physics and calculus. Look at our own brains – only analogies, I'm aware – and think about how much a child can infer about what will happen when you throw a ball at them. Are they 'calculating' trajectories? Not consciously. That would be too slow, regardless. Their brains have become wired through experience and evolution. Formally explaining why the ball will do what it does? Well, that takes years of math and physics education. Bottom line: in the months and years after a model comes out, research tends to uncover all sorts of things happening inside them that are unexpected. Best thing is to approach it with a beginner's mind, and say, "hey, it's doing _something_ interesting – let's try and understand what that is," rather than just forcing everything to conform to our existing worldview.

This is important, even though the Twitter part is useless. Here's the paper, "Revisiting Feature Prediction for Learning Visual Representations from Video"[1] A big problem with machine learning so far has been the lack of some underlying, more abstract model of the subject matter. I don't understand much of this, but apparently something called "representation", a purely mathematical concept, is able to help with this.[2] This turns some kinds of abstract problems into linear algebra problems, which means matrix operations, the things GPUs do so well, help. Code is available, from links in [2]. Would someone comment on this, please. Is this the beginning of a huge breakthrough? One where, instead of working in text token space or image pixel space, it's possible to work in an automatically generated more abstract space? [1] https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/427986745_768441298640104_1604906292521363076_n.pdf?_nc_cat=103&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=Lpq5IeF5ftUAX9eazCc&_nc_ht=scontent-sjc3-1.xx&oh=00_AfA9FKLM1Sfp86PgoyfCHg3UmLOH8stjie1QLOICKMqUkA&oe=65D69EB1 https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/427986745_... [2] https://en.wikipedia.org/wiki/Representation_theory https://en.wikipedia.org/wiki/Representation_theory

darthoctopus 3y ago
The "representation" in "representation theory" that the wikipedia article refers to is a term of art, referring to a class of one-to-one maps between groups (sets endowed with a binary operation with specific properties) and square matrices, in such a way that matrix multiplication preserves the properties of this binary operation between the original group elements being mapped to matrices. There is no mention of this sort of algebraic operation whatsoever in the paper you have linked, which, as far as I can tell, uses the word "representation" only in a nontechnical descriptive sense. I believe you might be misunderstanding the nature of this breakthrough.
esafak 3y ago
https://en.wikipedia.org/wiki/Feature_learning https://en.wikipedia.org/wiki/Feature_learning

givemeethekeys 3y ago

I can’t tell if his comments came before or after the latest OpenAI product announcement shoeing off what looks like generative video.

ghayes 3y ago
The original video: before, this tweet: after.
pixelesque 3y ago
World Government Summit was from 12-14 February 2024 in Dubai, so either a day before or at the same time, depending on timezones...

auggierose 3y ago

Just seeing an error message, and cannot sign in as well. Am I the only one?

auggierose 3y ago
Cleared the twitter web cache. Shitty software.

thriftwy 3y ago

[flagged]

2devnull 3y ago
Worse? It’s more direct.
- thriftwy 3y ago
  It's more creepy.
littlestymaar 3y ago
Love the analogy, stealing it.
subtra3t 3y ago
Original content of comment, in case it gets flagged: --- It is a huge stretch to assert that successful movie directors "understand the physical world". They do a worse thing: understand the human psyche. --- Unrelated but I really don't like these analogies. They use clever words to try to hand wave some vague ideas into existence, trying to appear smart but never really succeeding.

preommr 3y ago

More like: "Let me clear a huge misunderstanding here by using poorly-defined terms and cramming niche complex ideas elaborated elsewhere into this tweet." Someone correct me, but it seems like he's saying: 1- generative models don't understand the real world 2- generative models that work off of just pixels are more expensive and less useful than a model that represents the contents of the frame with abstract representations, particularly when related to higher-level actions like applying logic to a scene and not just interpolating between frames. 3- V-JEPA is such a model, and it performs particuarly well against generative when combining those representations as inputs with a much more easily trained and small model trained on specific tasks.

vasco 3y ago
I'm nowhere near an expert but it seemed like he was claiming true understanding of latent space is needed for generating coherent continuations, but Sora demo already has longish videos that are coherent. It's hard for me not to think this is someone just trying to still be right when they are wrong, but I may misunderstand.
- justanotherjoe 3y ago
  The way i understand it is that Sora is mostly just 'moving pictures' with no rhyme or reason. Yann Lecun is interested in videos that tell a 'story', with cause and effect. Like a magician putting his hand into a top hat and pulling out a rabbit, kind of video.
  - throw310822 3y ago
    Yeah, the same way ChatGPT is only predicting the next word with no rhyme or reason. However to actually predict the next word so that the entire sentence makes sense and is relevant for the context (e.g. answers a question) you probably must be actually understanding the meaning of the words and the language and have a world model. I can't imagine a NN moving around pixels in the shape of a cat with no understanding whatsoever of what a cat is, what it can do, how it is supposed to move, what it wants, etc.

deleted 3y ago

[deleted]

justanotherjoe 3y ago

"Furthermore, generating those continuations would be not only expensive but totally pointless." Why would it be pointless? I think there are many creative uses of video continuation model, considering the amount of control it gives you.

vasco 3y ago
If we cut all the videos in the world in half, half of the videos will be continuations. Or another way of saying, all video is continuation, so I'm with you, not sure why it would be pointless. In fact, a continuation based workflow with prompting seems like the easiest way to get a specific effect.
- timschmidt 3y ago
  Quick, feed in season 1 of Firefly.
  - stavros 3y ago
    As opposed to season 2? :(
profsummergig 3y ago
It's not pointless. It can potentially generate world simulations. We might be inside one such long video.

radarsat1 3y ago

The main thing I'm confused about by these comments is that as far as I understand, the Sora model (and many others like it) performs the diffusion process in latent space, and then translates this to pixels. So it's strange to me to claim that it doesn't have an abstract representation. But, maybe the latent space of a diffusion-VQVAE pipeline is fundamentally different from that of JEPA, I haven't read the relevant papers for that. Curious if someone could explain if they are different ideas of representation.

benreesman 3y ago
The claim, with admittedly limited consensus, is that a difference in degree becomes a difference in kind: we’ve known “forever in AI years” that restricting the inputs, and loss, and parameter dimensionality is right up there with the underlying architecture in terms of the properties those latent geometries exhibit. It’s even kind of a meme: the fingers problem (which neither Sora nor V-JEPA partisans seem to be claiming is “solved”). I’m a little oversubscribed at the moment so I haven’t downloaded the weights and played around with V-JEPA. The fact I’m pointing out that I could should make it pretty clear which way I lean on this: everyone wants to make money, let’s be real, but some seem resigned to “infinite, government-enforced monopoly via OpenPhilanthropy bribery” isn’t going to fly, so let’s at least prove how the sausage is made. It’s fairly uncontroversial that things in the neighborhood of a “bottlenecked” VAE are often forced to exploit structure if it’s there. This (claimed) result is about a way to exploit more of this structure/symmetry economy (greedy-ish optimizer) to pull the latent representations into yet a higher “effective regime” than yet demonstrated, with excellent properties around machine economics. Representation learning isn’t new (though LeCun is an acknowledged pioneer in it), and constraints as a powerful tool isn’t new (causal making in attention architectures shouldn't ruffle many feathers). Self/semi/unsupervised learning isn’t novel. But likewise, not are they k-equivalent synonyms: Dean et al. were distinguishing between Continuous Bag of Words and skip-gram in word2vec in 2013. But it does cool shit (I like the I-JEPA reconstructions as a go to slide deck raster), and it’s available weight. I think the truth shall soon emerge.

bonzaidrinkingb 3y ago

I feel LeCun got roped in debating the likes of Marcus and Yudkowsky. This has made his arguments lose nuance and become rigid. I also can't escape the feeling that if Facebook was tuned into Transformers, they would have shipped earlier, so there must have been some resistance or underestimation that's now repeated "They can't reason", "They can't plan", "They can't understand the world", "They are a distraction / side road to AGI". It is kind of ironic that researchers who claim LLMs lack adaptive intelligence seemingly refuse to adapt their intelligence to LLMs. If even GPT-3 can find logical holes or oversimplification in your arguments about GPTs, at one point this starts becoming embarrassing and unbecoming. > The generation of mostly realistic-looking videos from prompts does not indicate that a system understands the physical world. While arguably true, it also does not indicate that a system does not understand the physical world (reflections, collision detection, gravity, object permanence, long-term scene coherence, etc.). If LeCun wants to argue it does not understand the physical world, he should do so directly. Not attack something that is not directly stated, but rather convincingly and tentatively demo'd (I myself find it hard to argue that a system that generates novel pond reflections has not memorized/stored in weights some generalization program to apply to realistic scene generation). This demo shows it is not even a wild prediction to guess that soon (consumer tech) we will be able to discuss visual scenes with conversational AIs.

PoignardAzur 3y ago
> long-term scene coherence, FWIW none of the video models released so far demonstrate any object coherence whatsoever, which suggests they don't have the higher level capabilities you mention yet. In Sora, as soon as an object is obstructed by an obstacle or goes offscreen, it's likely to disappear or be radically transformed.
- bonzaidrinkingb 3y ago
  You've seen the demos of a couple holding hands and walking, or the museum shots where all the paintings maintain coherence, or a woman temporarily obscuring a street sign. Or you haven't seen those demos. Either way...

asimpletune 3y ago

I think one of the things that needs to be said about Sora is the videos that examples that we've seen have been impressive, but this is also how most of OpenAI's examples have been on marketing pages. What matters is when you yourself are actually able to use it. In the past, it was only when I've actually been able to try the tech myself that I've been able to see how successful (or unsuccessful) they really are for doing real work.

oldpersonintx 3y ago
No one is claiming Sora is a finished product HN gets hung up on damning things that aren't perfect _right now_ You also see this with FSD...it isn't perfect today, so HN writes it off forever Sora is a demo and a teaser of what will be a useful polished tool in three years, that's all
- pests 3y ago
  No, we've been writing off FSD for the last 6+ years it's been promised as "coming soon" People have bought, owned and used, and sold their Tesla w/ FSD without once being able to use the feature. Its not even any better or less relaxing - instead of driving you babysit a black box.

codeflo 3y ago

I mean, there's a point here. Even in the videos OpenAI shows as "successes", you can spot uncanny coherence mistakes. There's one where a forklift morphs into one that's turned 90 degrees. The previous frame looks a bit ambiguous about the direction the squarish shape was pointing at, and somehow what happens for several seconds before that wasn't a strong enough signal for the model. And in the "failure" videos, we see objects merging and splitting in similar ways that DALL-E inserts extra arms into scenes with groups of people. To me, it's clear that there's no real global understanding yet. Whether this person's preferred ML approach solves that, I'm not knowledgeable enough to tell.

bamboozled 3y ago
Someone on here was claiming Sora understands physics, while I was watching steaming frozen snow, the lady in red skating down the street in Tokyo on wet concrete and a bunch of people having lunch next to miniature people visiting a mini market stand made from a bike. Definitely felt like something was off to me.

awaythrow999 3y ago

It's impossible to access this content (even more so now that nitter is dead). Should HN continue driving traffic to a site that is entirely inaccessible?

zmgsabst 3y ago
I don’t understand: This link opens the post for me, while paywalled news is semi-routinely posted. What’s the objection?
- pavlov 3y ago
  The difference is that paywalled news sites require a login to access content that they themselves produced and published, while Twitter/X now requires a login to access content written by other people. In other words, paywalled news is a first party content provider, but Twitter is just a third party who inject their mandatory login between me (the reader) and the writer so they can make some profit by showing ads on the content. That's fine, but authors should consider if they want to publish on a site like that when there are so many other options.
- Mistletoe 3y ago
  Sometimes Twitter links won’t open for me now without being logged in. I’ve loved the change because it has helped me not use Twitter at all. This one opened for me though.
yreg 3y ago
hn should drive traffic wherever the content is published / mirrored. If there are multiple options, choose the most public one. We shouldn't censor ourselves and not discuss information that was posted on twitter.
- awaythrow999 3y ago
  Other paywalled content at least can be opened with archive.md and friends. Usually people here kindly post a link for convenience to archive. With twotter that no longer works. So this type of content means that in order to follow the discussion I must also be a twitter subscriber. Maybe people could consider first creating an archive.org snapshot that is then linked instead. Would be the kind thing to do and help the rest of us.

advael 3y ago

I feel like LeCunn often gets it half right and then gets overzealous/has a conflict of interest. I think there are good reasons to be less than perfectly optimistic that OpenAI's claims will bear out (The kind of errors SORA makes seem to imply that "simulation" may be a stretch in my opinion, I talked about this a little in the thread about that announcement and honestly when I first read this post I thought he hit the nail on the head until about the fifth sentence), but I don't think there's strong evidence to suggest that they definitely can't yet, and this seems more like a sales pitch for the model he's currently working on than an expert making a particularly compelling case via better understanding of the theory or practice of this thing Like, in general I think there's a lot of hype around AGI and that skepticism toward OpenAI's claims isn't completely unwarranted, but the amount of public attention on the topic lately has caused everyone to commit to very hard lines that lack nuance about the implications of various advances. It's in some ways cool that AI is no longer just an academic curiosity, but it makes for a lot of nonsense to slog through, even from top researchers

dkjaudyeqooe 3y ago

OpenAI willfully fuels hype of all sorts and lets people extrapolate without basis or limit, because it's hugely profitable for them. LeCun is the voice of reason trying to point out the limitations of the current technology and fundamental questions that need to be answered. Until there is something more than cute demos, like an actual path forward that can implement what people are hyping, or better still working examples, it's all speculative nonsense.

bamboozled 3y ago
LeCun is the voice of reason trying to point out the limitations of the current technology and fundamental questions that need to be answered. I think this is the more likely true, but less popular take as well.

cactusplant7374 3y ago

It's interesting how far behind xAI already is in the current market. Their current advantage of having a corpus of text isn't even relevant.

Sai_ 3y ago
Just a random thought - maybe Musk was/is so hell bent on removing censorship because he wants more robust data, covering a larger array of thoughts, for his AI plans.

ml-anon 3y ago

LeCun is such a hack and is guilty exactly the same hype as OpenAI. Firstly his insistence on “self supervised learning” which is just a wrong and unhelpful rebranding of existing methodologies. Followed by talking about VicREG as if it’s a meaningful contribution and not just hacked together crap which is not only theoretically unfounded but plain nonsensical. Followed again by his “JEPA” work which again is just a rebranding of other works like BYOL and data2vec. Its so frustrating to see this guy pop up and try to claim credit for an entire field of representation learning which he hasn’t made a single meaningful contribution to in decades.

optimalsolver 3y ago
>“self supervised learning” which is just a wrong and unhelpful rebranding of existing methodologies What are those methodologies?
- loveparade 3y ago
  Self-supervised learning is such a trivial concept that it doesn't need a new term. It's just marketing/PR speech to attract attention. You could just call it "generating labels from data" All these stupid new terms of trivial concepts confuse people who are new to the field. That's nothing new though, research has always been like this, people love to make up new phrases that they can own and sell as novel ideas to reviewers, even if they're just trivial renamings. It's nothing more than a PR game.
  - optimalsolver 3y ago
    If people didn't consider it a useful pointer to a certain set of ideas, it wouldn't have caught on in the ML community. That's how language works.
- ml-anon 3y ago
  In the 90’s and 2000’s we called it unsupervised learning. In fact there is a direct formal equivalence between many so-called self supervised learning techniques and matrix factorization. And literally no one would claim that matrix factorisation is anything other than unsupervised. Similarly Yann makes such a big deal about not doing contrastive learning when contrastive methods and his JEPA nonsense can also be shown to be formally equivalent. He’s a grifter like the rest of them. There’s a reason why he basically holds no power at Meta and hasn’t been in charge of AI research there in a long time.

deleted 3y ago

[deleted]

Havoc 3y ago

>does not indicate that a system understands the physical world. That's a bit like saying the chatbots aren't actually intelligent. Sure, but there is at least a plausible illusion of it & even that has usefulness. The same applies to physical world. e.g. Look at the reflections in the SORA demo. They're not right, but they're also not entirely wrong either. That to me suggests usefulness in approximating the physical world

neonate 3y ago

https://archive.md/Jysf2 https://archive.md/Jysf2

fancyfredbot 3y ago

It's a shame that Yann didn't explicitly say what the misunderstanding was or clear it up. I think he's claiming that the misunderstanding is that SORA understands the physical world. He goes on to say that generating the next frame conditional on some action is much harder than generating an entire plausible video. This just doesn't make sense to me. Every frame in a plausible video clearly needs to be conditioned on the actions shown in previous frames. The tweet is mostly incoherent and I'm left assuming he's upset about SORA and wants to say his ideas are/were better but hasn't managed to express why in a meaningful way.

makomk 3y ago
As I understand it, diffusion-based video generation models simply are not casual in this way. They work by modifying the previous frames in the video to be consistent with future frames just as much as they do later frames to be consistent with earlier ones. That's why Yann LeCun can argue that they do not have to be able to generate plausible continuations of a real video, just generate some arbitrary sample from the space of plausible-looking videos, and that the latter does not imply the ability to do the former. It's also why it's not possible to just generate videos of arbitrary length and lots of VRAM is required to create even a relatively short clip.
- fancyfredbot 3y ago
  Thank you. That would make sense. Perhaps Yann's target audience are supposed to know this already, but your explanation actually cleared up a misunderstanding for me.

gdubs 3y ago

I quote Zen Mind Beginner's Mind a lot: "In the beginner's mind there are many possibilities, but in the expert's there are few." It's like, the smarter someone gets, the more it trips them up when it comes to questions like the one Yann is trying to address. Because, "world modeling" doesn't have to mean that the system understands advanced physics and calculus. Look at our own brains – only analogies, I'm aware – and think about how much a child can infer about what will happen when you throw a ball at them. Are they 'calculating' trajectories? Not consciously. That would be too slow, regardless. Their brains have become wired through experience and evolution. Formally explaining why the ball will do what it does? Well, that takes years of math and physics education. Bottom line: in the months and years after a model comes out, research tends to uncover all sorts of things happening inside them that are unexpected. Best thing is to approach it with a beginner's mind, and say, "hey, it's doing _something_ interesting – let's try and understand what that is," rather than just forcing everything to conform to our existing worldview.