6 ms·
V-JEPA 2 world model and new benchmarks for physical reasoning
- deleted 1y ago[deleted]
- artificialprint 1y agoThrow ARC-AGI 2 at it!
- jadbox 1y agoI suspect it wouldn't help too much. This model is meant for physics-based world modeling, while nearly all the problems in ARC are symbolic reasoning.
- artificialprint 1y agoI'd say world modeling can provide the foundations from which symbolic reasoning can emerge, after all this is how we (humans) learn it too. There are a lot of tasks in arc that are grounded in simple physics
- littlestymaar 1y ago> I'd say world modeling can provide the foundations from which symbolic reasoning can emerge, after all this is how we (humans) learn it too As usual comparisons with humans provide little practical insight for what's achievable with ML. Humans don't have to learn everything from scratch like ML models do, you aren't expecting ML models to learn language out of a few thousands of tokens just because humans can, so similarly you shouldn't expect neural networks to learn reasoning from world interaction alone.
- falcor84 1y agoYes, ARC-AGI 2 seems to game a lot of challenges that involve a (projection of) gravity and collisions, so I'd be quite interested in seeing whether it would generalize.
- ldjkfkdsjnv 1y agoLeadership at meta is dropping the ball with these non llm ai model sidequests
- jadbox 1y agoLLMs where once a side quest. I hope meta invests more in alternatives as maybe we'll find something better. If not, then meta just loses a bit of R&D budget. They are still heavily invested in regular LLM development, so it's not like they are trading one for the other.
- linguistbreaker 1y agoI strongly agree. FAANG has the money to do the research. LLMs are far from intelligent - AGI will require a number of other advances.
- energy123 1y agoIs this a sarcastic compliment? Diversity in research agendas is very important for pushing forward the frontier even if it's not good for the company investing in the high risk research. Good job, to an otherwise toxic company.
- rvz 1y agoAI research is more than just LLMs.
- TheAceOfHearts 1y ago> With these visual subgoals, V-JEPA 2 achieves success rates of 65% – 80% for pick-and-placing new objects in new and unseen environments. How does this compare with existing alternatives? Maybe I'm just lacking proper context, but a minimum 20% failure rate sounds pretty bad? The paper compares their results with older approaches, which apparently had something like a 15% success rate, so jumping to an 80% success rate does seem like a significant jump. If I'm reading the paper correctly, the amount of time required to compute and execute each action went down from 4 minutes to 16 seconds, which also seems significant. Having to specify an end goal as an image seems pretty limited, but at least the authors acknowledge it in the paper: > Second, as mentioned in Section 4, V-JEPA 2-AC currently relies upon tasks specified as image goals. Although this may be natural for some tasks, there are other situations where language-based goal specification may be preferable. Extending the V-JEPA 2-AC to accept language-based goals, e.g., by having a model that can embed language-based goals into the V-JEPA 2-AC representation space, is another important direction for future work. The results described in Section 7, aligning V-JEPA 2 with a language model, may serve as a starting point. I think it would be interesting if the authors answered whether they think there's a clear trajectory towards a model that can be trained to achieve a >99% success rate.
- ricardobeat 1y agoIt’s important to keep some perspective: there are zero robots in the wild, at the moment, that use a world model to work on tasks they weren’t specifically trained on. This is cutting edge research and an 80% success rate is astonishing!
- gyudin 1y agoThey don’t use it because it’s unsafe and potentially life threatening lol
- dghlsakjg 1y agoPlenty of things are unsafe and potentially life threatening, including machines with pre-programmed routines that we use today. We already have robots with limited intelligence interacting safely with humans in workplaces. This learning technology didn't exist until this moment in time. That probably has more to do with why no one is using it in the wild.
- fidotron 1y agoYou have to wonder if the model is going to end up recreating Verlet integration in there somewhere, or if it's generating a pile of those optical acceleration cancelation type heuristics in neural net form. It's one of those ideas I've had around for a while that if you fused decent object tracking with an understanding of Verlet integration you should, in principle, start being able to measure all sorts of physical quantities quite easily.
- nlitened 1y agoI imagine that Russian-speaking team members had fun with naming the model V-JEPA
- Tiberium 1y agoFor the curious: "жопа" (which "JEPA" sounds like) means "ass" in Russian. Also V ("В") means "in" (although if we get into specifics, the casing would need to be "жопу" or "жопе" depending on the context)
- koakuma-chan 1y agoAlso the video thumbnail: J.E.P.A.
- deleted 1y ago[deleted]
- jcelerier 1y ago> That kind of physical intuition isn’t something adults obtain after years of education—young children develop this intuition by observing the world around them before they can even speak in full sentences. I mean, it still takes them much more time than it takes to train even the largest LLMs we use (a couple months)
- lukan 1y agoBut they use way less energy for it.
- dist-epoch 1y agoIn wall clock time. If you count in input tokens/pixels, humans learn with orders of magnitude less input data.
- logicchains 1y agoThat's not true at all; the amount of audiovisual data a human is exposed to in even just one year is incredibly vast. Over sixty frames per second, sixteen hours per day gives over a trillion frames per year, and each frame at such a high resolution would be hundreds of tokens.
- dist-epoch 1y agoLet's take your numbers: Human: 1000 tok * 60 * 86400 * 365 = 2 Trillion tokens / year GPT-4: 13 Trillion tokens Llama-3: 15 Trillion tokens
- naasking 1y agoThat's vision only mind you, no consideration of sound, taste, touch, and interoception. And GPT-4 is far more fluent and knowledgeable than a 6-7 year old where the total token count matches by this assessment (though I'm skeptical of the numbers given).
- iamnotagenius 1y ago
- momojo 1y agoWhy is Meta investing into this research? What's the potential payoff?
- dyauspitr 1y agoPhysical robots as impressive as LLMs?
- aaroninsf 1y agoThe goal is a a Large Phenomenological Model. A good definition of "real AGI" might be, a multimodal model which understands time-based media, space, and object behavior, and hence true agency. Phenomenology is the philosophy of "things as they seem," not "knowledge (words) about things." Seem to our senses, not understood through language. LLM of course trade in language tokens. We can extend their behavior with front ends which convert other media types into such tokens. But we can do better with multimodal models which are trained directly on other inputs. E.g. integrating image classifiers with language models architecturally. With those one can sort of understand time-based media, by sampling a stream and getting e.g. transcripts. But again, it's even better to build a time-base multimodal models, which directly ingests time-based media rather than sampling. (Other architectures than transformers are going to be required to do this well IMO...) The bootstrapping continues. This work is about training models to understand world and object properties by introducing agency. Significant footnote: implicitly models trained to interact with the world necessarily have a "self model" which interacts with the "world model." Presumably they are trained to preserve their expensive "self." Hmmmmm.... When we have a model that knows about things not just as nodes in a language graph but also how such things look, and sound, and moves, and "feel" (how much mass do they have, how do they move, etc.)... ...well, that is approaching indistinguishable from one of us, at least wrt embodiment and agency.
- deleted 1y ago[deleted]
- DesiLurker 1y agopossibly with their investment into AR/VR and gaming they may see a pathway to creating 'physical intelligence' and tap into a much bigger untapped market. I mean isn't Robotaxi the main carrot Musk's been holding in front of tesla investors for decade or so. physical robots may provide a more 'incremental fault tolerant' path to application of AI.
- cubefox 1y agoI think the fundamental idea behind JEPA (not necessarily this concrete Meta implementation) will ultimately be correct: predicting embeddings instead of concrete tokens. That's arguably what animals do. Next-token prediction (a probability distribution over the possible next tokens) works well for the discrete domain of text, but it doesn't work well for a continuous domain like video, which would be needed for real-time robotics. For text, with a two-byte tokenizer you get 2^16 (~65.000) possible next tokens, and computing a probability distribution over them is very much doable. But the "possible next frames" in a video feed would already be an extremely large number. If one frame is 1 megabyte uncompressed (instead of just 2 bytes for a text token) there are 2^(8*2^20) possible next frames, which is far too large a number. So we somehow need to predict only an embedding of a frame, of how the next frame of a video feed will look approximately. Moreover, for robotics we don't want to just predict the next (approximate) frame of a video feed. We want to predict future sensory data more generally. That's arguably what animals do, including humans. We constantly anticipate what happens to us in "the future", approximately, and where the farther future is predicted progressively less exactly. We are relatively sure of what happens in a second, but less and less sure of what happens in a minute, or a day, or a year.
- abraxas 1y agoBut how do you go from predicting embeddings (which could be thought of as a type of lossy compression of the original data) back out to something usable, say a sequence of image/video tokens or a sequence of robot actions?
- cubefox 1y agoA robot model would need to constantly convert the prediction (an embedding) of the future observations, together with a "plan" of what the robot tries to achieve, into an action. Into some kind of movement which takes both the action plan and the predicted sensory data into account. That's very much an unsolved problem, and I don't know how far Meta is along that path. Not very far, I assume.
- NitpickLawyer 1y ago
- siavosh 1y agoDoes someone know how the "semantic" embeddings are learned? That seems like perhaps the main technical challenge here.
- gglon 1y agoFrom the paper, section 2.1: minimize_θ,φ,Δ ||P_φ(Δ, E_θ(x)) - sg(E_θ'(y))||_1 where y - full video, x - masked video, E_θ(.) - learned encoder (semantic embedding), P_φ(.) - learned predictor, Δ - learned mask (which patches in a video where dropped), sg(.) - stop gradient to prevent change, gradient propagation in E_θ'(.), which in turn is an exponential moving average of E_θ(.) ie. θ'_new <- τ θ'_old + (1-τ) θ. So the loss is applied only to the predictions of the masked patches while the encoder of full video follows the learned one. This asymmetry in learning prevents collapse of the encoder to a trivial constant.
- iLoveOncall 1y ago"World model" and "physical reasoning" is such a lie. Those models don't have any understanding of physics, they just regurgitate what they see in their vision-based training set, just like any image or video generation model does. Monkey see other monkey cannot go through wall, monkey don't try go through wall.
- rayboy1995 1y ago> Monkey see other monkey cannot go through wall, monkey don't try go through wall. I mean... we are just monkeys. Did we not learn this way when we were younger?
- RollingRo11 1y agoAgreed! A really young child has no notion of "physics". They are learning through experience and observation. These models/robots aren't superintelligent by any means, but "Monkey see other monkey cannot go through wall, monkey don't try go through wall" isn't far off from how some animals/humans "learn".
- deleted 1y ago[deleted]
- smokel 1y agoI think you are misinterpreting the terminology. Of course these models are not understanding physics in the way a physicists or a mathematician would. But they do form a model of the world that can be used for forecasting and reasoning, in a way possibly not much unlike how humans and other animals operate when interacting with the physical world.
- seydor 1y agophysics is phenomenological. the model sees phenomena
- dghlsakjg 1y agoYou don't need to have taken a single physics class to be good at pool...
- rar00 1y agothe robot arm demonstration video jumps at the 00:28s mark...
- aew2120 1y agoInterestingly, a small company called Ogma already did something very similar back in 2021 (on an embedded system, no less). This (https://ogma.ai/2021/07/unsupervised-behavioral-learning-ubl-revisited/ https://ogma.ai/2021/07/unsupervised-behavioral-learning-ubl...) is a description/video of how they got a small RC car to predict the next frame of its video feed given the action it was about to take, and thereby made the car navigate to a given location when fed with a still frame of that location (all this with online learning, and no backprop). Instead of vicreg, they induced their latent state with sparse auto-encoding. Also they predicted in pixel, as opposed to latent, space. The white paper describing their tech is a little bit of a mess, but schematically, at least, the hierarchical architecture they describe bears a strong resemblance to the hierarchical JEPA models LeCunn outlined in his big paper from a few years ago. A notable difference, though, is that their thing is essentially a reflex agent, as opposed to possessing a planning/optimization loop.
- kadushka 1y agoThe ideas at Ogma are inspired by Numenta's work.
- concrete_head 1y agoJust wanted to say thank very much for sharing this. Over the last few months I've been inventing this almost exact approach in my head as a hobby without consciously knowing it had already been done. I love their little RC car demo.