6 ms·
Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meani
by vatsachak 4mo ago
Are you joking? They spend billions of dollars training LLMs to get a 7.8% on arc agi 3 whereas DINO models are near sota in image classification, provide meaningful embeddings to the point where image segmentation is just PCA. The spend on DINO cannot be more than five million (correct me if I'm wrong)
JEPA is just getting started
- esafak 4mo agoASI is going to be here by the time Lecun gets started.
- redactsureAI 4mo agoDINO is a transformer model?
- vatsachak 4mo agoJEPA can use a transformer, and DINO does so yes
- deleted 4mo ago[deleted]
- Tenoke 4mo agoHis main anti-LLM predictions have been consistently either wrong or misleading. There's many ways to skin a cat so you can probably do something with a JEPA approach as well, but I doubt he actually catches up to having agents on the level of where Anthropic/OpenAI will be at any point.
- onlyrealcuzzo 4mo agoHis main LLM predictions have almost nothing to do with Arc AGI... What exactly was he dead wrong about that is proven by any of this? GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying. He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3. He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value. Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.
- Tenoke 4mo ago>He's merely said they don't think He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.
- onlyrealcuzzo 4mo ago> He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000. What things specifically and when?
- Tenoke 4mo agohttps://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.
- onlyrealcuzzo 4mo agoThat does not at all look cherry picked or taken out of context... LeCun's ideas cannot be reduced to a 6 second clip... You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...
- Tenoke 4mo agoYou can watch the whole Lex Friedman interview, it's on youtube. It's not out of context at all. He goes on about how LLMs will never be able to do things that they do trivially. And he has just doubled down for years. Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.
- typon 4mo agoDinoV3 paper: https://arxiv.org/pdf/2508.10104#page=36 https://arxiv.org/pdf/2508.10104#page=36 "we use a rough estimate of a total 9M GPU hours" From CoreWeave, at current prices (~$2.46/hr spot to ~$6.16/hr on demand) would correspond to $22M–$55M. The dataset is really where the cost is though - they used LVD-1689M - 1.6B images of curated web data from roughly 17B instagram images. This probably cost a huge amount of hours in human annotation, compute for algorithmic filtering, etc and not to mention probably a 20-50 person team working on this model. You might want to change assumptions about how expensive these models are.
- vatsachak 4mo agoThanks for the correction on the order of magnitude for the whole training process. The 9M GPU hours includes the DINO v2 inference used in order to curate the data set. The final training run used like 300000 dollars of compute. Unfortunately we don't know how much RLVR + Agent training costs these companies. I'm just gonna say it's in the hundreds of millions, because they are supposedly making billions of profit on inference yet making billion dollar losses
- ainch 4mo agoYann is a big SSL guy but I don't think he was involved in the original DINO - he's not listed as a co-author or anything.
- vatsachak 4mo agoDINO was created independent of JEPA but uses a similar principle of self supervised learning through minimizing the prediction error of a latent. The difficulty in predicting a latent is so called "collapse"; the embedding neutral network can always output the zero vector and this would predict the output correctly. There are different ways to solve this, DINO uses two different models - a teacher and a student and LeCunn uses an explicit term against collapsing to a single output. Yann mentions DINO in his talks