2 ms·

> How is Astra a step change in computer use and spatial reasoning to the extent it can play games and paint good looking stuff with e.g canva ? Presumably bec

> How is Astra a step change in computer use and spatial reasoning to the extent it can play games and paint good looking stuff with e.g canva ? Presumably because of pre-release training, because some alien outside of the model, armed with the reinforcement learning algorithm, came in and programmed its weights. > The architecture that keeps delivering results isn't slowing down ? You don't say Sure, nothing wrong with that, as long as you don't misrepresent the limitations of the approach. > There's no shortage of people, even researchers, who for one reason or the other are convinced we are in need of some paradigm shift. > But guess what? Talk is cheap. You beat the current paradigm or you don't. Do you seriously think that Meta never scaled JEPA ? The idea has not been taken very far, so what is there to scale? It's not a complete cognitive architecture. So far it's also been using a pre-trained transformer as the learning component, which makes it of limited interest. The animal intelligence approach, even in it's most fledgling form (that you would apparently dismiss), requires a complete agentic architecture, including new learning algorithms and generative behavior, before it can be compared to LLMs. We know that, done right, our brain architecture is more capable than an LLM, so even if any hypothetical attempts to reproduce it were not highly performant, we know that the idea itself is sound. You might compare with Uszkoreit's initial poor-performing implementation of his new language model architecture - should he have given up?

famouswaffles 1mo ago
>Presumably because of pre-release training, because some alien outside of the model, armed with the reinforcement learning algorithm, came in and programmed its weights. They didn't program anything. They gave it data at best. >The idea has not been taken very far, so what is there to scale? LeCun was the head of Meta AI for over a decade and his baby that he keeps harping on about wasn't taken very far ? Come on. You're smarter than that. It went the way all the alternate architectures have gone since the transformer, a sidegrade at best, probably not even that. >You might compare with Uszkoreit's initial poor-performing implementation of his new language model architecture - should he have given up? What are you talking about? There was no poor performing implementation of transformers that was published that he needed to push through. Are you talking about self attention experiments before the finished transformer? That's literally just research. And if he languished on that for a decade then yeah I'd tell him to probably look at something else, but of course he didn't.
- HarHarVeryFunny 1mo ago
  You regard the LLM post-training process as "giving it data" ?! Have you looked at all the published JEPA research both while LeCun was at Meta, and since (up to and including the latest AdaJEPA from June)? Please enlighten us as to exactly which line(s) of research you think were "scaled" at Meta, and then tell us which of these constituted anything even remotely resembling a complete testable intelligence? FYI, it's been a long time since FaceBook/Meta even had a single head of AI. Since 2018 it has been split into two groups, FAIR and Generative AI, with LeCun being in the FAIR group, not as head, but as Chief AI scientist. LeCun only invented JEPA in 2022 (shortly after FaceBook became Meta), first writing about it in his "A Path Towards Autonomous Machine Intelligence" paper, perhaps unhappy with the work of the GenAI group, which he had no control over, that presumably was getting all the compute. https://openreview.net/pdf?id=BZ5a1r-kVsf https://openreview.net/pdf?id=BZ5a1r-kVsf > What are you talking about? I was referring to Uszkoreit's personal telling (on YouTube) of the origin story of the Transformer, his motivations with the design, his initial personal failure to implement his idea in a performant enough manner to beat the current LSTM SOTA, and Noam Shazeer then throwing the kitchen sink at it and eventually coming up with the Transformer design.
  - famouswaffles 1mo ago
    >You regard the LLM post-training process as "giving it data" ?! Yeah. Presumably, lots of synthetic data is being generated, experiments being run, but post-training is still a largely automated process. >Have you looked at all the published JEPA research both while LeCun was at Meta, and since (up to and including the latest AdaJEPA from June)? Please enlighten us as to exactly which line(s) of research you think were "scaled" at Meta, and then tell us which of these constituted anything even remotely resembling a complete testable intelligence? I have. My point isn't that his ideas are trash or that he should stop working on them. My point is it's not "gone very far" because he's taking it as far as he can, which isn't very far. He's not had a lack of influence, resources or will, either from his time at Meta or now with his billion dollar startup. He's had far more of it than most, if anything. That there's not much to show for it so far is not for a lack of trying. >I was referring to Uszkoreit's personal telling (on YouTube) of the origin story of the Transformer, his motivations with the design, his initial personal failure to implement his idea in a performant enough manner to beat the current LSTM SOTA, and Noam Shazeer then throwing the kitchen sink at it and eventually coming up with the Transformer design. So it's what I thought. This is just regular research unless an inordinate amount of time was spent on it and that's not the case.

famouswaffles 1mo ago

>Presumably because of pre-release training, because some alien outside of the model, armed with the reinforcement learning algorithm, came in and programmed its weights. They didn't program anything. They gave it data at best. >The idea has not been taken very far, so what is there to scale? LeCun was the head of Meta AI for over a decade and his baby that he keeps harping on about wasn't taken very far ? Come on. You're smarter than that. It went the way all the alternate architectures have gone since the transformer, a sidegrade at best, probably not even that. >You might compare with Uszkoreit's initial poor-performing implementation of his new language model architecture - should he have given up? What are you talking about? There was no poor performing implementation of transformers that was published that he needed to push through. Are you talking about self attention experiments before the finished transformer? That's literally just research. And if he languished on that for a decade then yeah I'd tell him to probably look at something else, but of course he didn't.

HarHarVeryFunny 1mo ago
You regard the LLM post-training process as "giving it data" ?! Have you looked at all the published JEPA research both while LeCun was at Meta, and since (up to and including the latest AdaJEPA from June)? Please enlighten us as to exactly which line(s) of research you think were "scaled" at Meta, and then tell us which of these constituted anything even remotely resembling a complete testable intelligence? FYI, it's been a long time since FaceBook/Meta even had a single head of AI. Since 2018 it has been split into two groups, FAIR and Generative AI, with LeCun being in the FAIR group, not as head, but as Chief AI scientist. LeCun only invented JEPA in 2022 (shortly after FaceBook became Meta), first writing about it in his "A Path Towards Autonomous Machine Intelligence" paper, perhaps unhappy with the work of the GenAI group, which he had no control over, that presumably was getting all the compute. https://openreview.net/pdf?id=BZ5a1r-kVsf https://openreview.net/pdf?id=BZ5a1r-kVsf > What are you talking about? I was referring to Uszkoreit's personal telling (on YouTube) of the origin story of the Transformer, his motivations with the design, his initial personal failure to implement his idea in a performant enough manner to beat the current LSTM SOTA, and Noam Shazeer then throwing the kitchen sink at it and eventually coming up with the Transformer design.
- famouswaffles 1mo ago
  >You regard the LLM post-training process as "giving it data" ?! Yeah. Presumably, lots of synthetic data is being generated, experiments being run, but post-training is still a largely automated process. >Have you looked at all the published JEPA research both while LeCun was at Meta, and since (up to and including the latest AdaJEPA from June)? Please enlighten us as to exactly which line(s) of research you think were "scaled" at Meta, and then tell us which of these constituted anything even remotely resembling a complete testable intelligence? I have. My point isn't that his ideas are trash or that he should stop working on them. My point is it's not "gone very far" because he's taking it as far as he can, which isn't very far. He's not had a lack of influence, resources or will, either from his time at Meta or now with his billion dollar startup. He's had far more of it than most, if anything. That there's not much to show for it so far is not for a lack of trying. >I was referring to Uszkoreit's personal telling (on YouTube) of the origin story of the Transformer, his motivations with the design, his initial personal failure to implement his idea in a performant enough manner to beat the current LSTM SOTA, and Noam Shazeer then throwing the kitchen sink at it and eventually coming up with the Transformer design. So it's what I thought. This is just regular research unless an inordinate amount of time was spent on it and that's not the case.