4 ms·

Per the author’s links, he warned that deep learning was hitting a wall in both 2018 and 2022. Now would be a reasonable time to look back and say “whoops, I wa

• otabdeveloper4 10mo ago
  > learning was hitting a wall in both 2018 and 2022 He wasn't wrong though.
• Ukv 10mo ago
  Even further back: > Yet deep learning may well be approaching a wall, much as I anticipated earlier, at beginning of the resurgence (Marcus, 2012) (From "Deep Learning: A Critical Appraisal")
• bgwalter 10mo ago
  Several OpenAI people said in 2023 that they were surprised by the acceptance of the public. Because they thought that LLMs were not so impressive. The public has now caught up with that view. Familiarity breeds contempt, in this case justifiably so. EDIT: It is interesting that in a submission about Sutskever essentially citing Sutskever is downvoted. You can do it here, but the whole of YouTube will still hate "AI".
  • Jyaif 10mo ago
    > in this case justifiably so Oh please. What LLMs are doing now was complete and utter science fiction just 10 years ago (2015).
    • deadbabe 10mo ago
      Not really. Any fool could have anticipated the eventual result of transformer architecture if pursued to its maximum viable form. What is impressive is the massive scale of data collection and compute resources rolled out, and the amount of money pouring into all this. But 10 years ago, spammers were building simple little bots with markov chains to evade filters because their outputs sounded plausibly human enough. Not hard to see how a more advanced version of that could produce more useful outputs.
      • free_bip 10mo ago
        I guess I'm worse than a fool then, because I thought it was totally impossible 10 years ago.
      • Workaccount2 10mo ago
        Any fool could have seen self driving cars coming in 2022. But that didn't happen. And still hasn't happened. But if it did happen, it would be easy to say: "Any fool could have seen this coming in 2012 if they were paying attention to vision model improvements" Hindsight is 20/20.
        • lisbbb 10mo ago
          Everyone who lives in the show belt understands that unless a self driving car can navigate icy, snow-covered roads better than humans can, it's a non-starter. And the car can't just "pull over because it's too dangerous" that doesn't work at all.
          • phil21 10mo ago
            That works fine. Self driving doesn’t need to be everything for all conditions everywhere. Give me reliable and safe self driving for Interstate highways in moderate to good weather conditions and I would be very happy. Get better incrementally from there. I live solidly in the snow belt. Autopilot for planes works in this manner too. Theoretically a modern airliner could autofly takeoff to landing entirely autonomously at this point, but they do not. They decrease pilot workload. If you want the full robotaxi panacea everywhere at all times in all conditions? Sure. None of us are likely to see that in our lifetime.
            • fragmede 10mo ago
              Btw that’s basically already here with http://comma.ai http://comma.ai.
        • deadbabe 10mo ago
          We definitely have self driving cars, people just want to move the goal posts constantly.
    • bgwalter 10mo ago
      Why would the public care what was possible in 2015? They see the results from 2023-2025 and aren't impressed, just like Sutskever.
    • lisbbb 10mo ago
      What exactly are they doing? I've seen a lot of hype but not much real change. It's like a different way to google for answers and some code generation tossed in, but it's not like LLMs are folding my laundry or mowing my lawn. They seem to be good at putting graphic artists out of work mainly because the public abides the miserable slop produced.
      • HDThoreaun 10mo ago
        My teams velocity is up around 50% because of ai coding assistants.
      • deleted 10mo ago
        [deleted]
    • absoluteunit1 10mo ago
      This. I’m under the impression that people who are still saying LLMs are unimpressive might just be not using them correctly/effectively. Or as Primagean says: “skill issue”
    • hattmall 10mo ago
      In what way do you consider that to be the case? IBM's Watson defeated actual human champions in Jeopardy in 2011. Both Walmart and McDonald's notably made large investments shortly after that on custom developed AI based on Watson for business modeling and lots of other major corporations did similar things. Yes subsidizing it for the masses is nice but given the impressive technology of Watson 15 years ago I have a hard time seeing how today's generative AI is science fiction. I'm not even sure that the SOTA models could even win Jeopardy today. Watson only hallucinated facts for one answer.
      • ben_w 10mo ago
        When Watson did that, everyone initially was very impressed, but later it felt more like it was just a slightly better search engine. LLMs screw up a lot, sure, but Watson couldn't do code reviews, or help me learn a foreign language by critiquing my use of articles and declination and idiom, nor could it create an SVG of a pelican riding a bicycle, nor help millions of bored kids cheat on their homework by writing entire essays for them.
• tim333 10mo ago
  The author is a bit of a stopped clock that who has been saying deep learning is hitting a wall for years and I guess one day may be proved right? He probably makes quite good money as the go to guy for saying AI is rubbish? https://champions-speakers.co.uk/speaker-agent/gary-marcus https://champions-speakers.co.uk/speaker-agent/gary-marcus
  • chii 10mo ago
    a contrarian needs to keep spruiking the point, because if he relents, he loses the core audience that listened to him. That's why it's also the same with those who keep predicting market crashes etc.
    • weatherlite 10mo ago
      Well the same can be said about non contrarians ...
      • enknee1 10mo ago
        The same can be said about hucksters of all stripes, yes. But maybe not contrarians/non-contrarians? They are just the agree/disagree commentators. And much of the most valuable commentary is nuanced with support for and against their own position. But generally for.
  • JKCalhoun 10mo ago
    I thought the point though was that Sutskever is saying it too.
  • jvanderbot 10mo ago
    Well..... tbf. Each approach has hit a wall. It's just that we change things a bit and move around that wall? But that's certainly not a nuanced / trustworthy analysis of things unless you're a top tier researcher.
    • espadrine 10mo ago
• jayd16 10mo ago
  If something hits a wall and then takes a trillion dollars to move forward but it does move forward, I'm not sure I'd say it was just bluster.
  • dwaltrip 10mo ago
    They didn't spend a trillion dollars to create GPT-3 in 2020... Gary Marcus is a mindless talking head "contrarian" at this point. He should get a real job.
• chubot 10mo ago
  I read Deep Learning: A Critical Appraisal ? in 2018, and just went back and skimmed it https://arxiv.org/abs/1801.00631 https://arxiv.org/abs/1801.00631 Here are some of the points Is deep learning approaching a wall? - He doesn't make a concrete prediction, which seems like a hedge to avoid looking silly later. Similarly, I noticed a hedge in this post: Of course it ain’t over til it’s over. Maybe pure scaling ... will somehow magically yet solve ... --- But the paper isn't wrong either: Deep learning thus far is data hungry - yes, absolutely Deep learning thus far is shallow and has limited capacity for transfer - yes, Sutskeyer is saying that deep learning doesn't generalize as well as humans Deep learning thus far has no natural way to deal with hierarchical structure - I think this is technically true, but I would also say that a HUMAN can LEARN to use LLMs while taking these limitations into account. It's non-trivial to use them, but they are useful Deep learning thus far has struggled with open-ended inference - same point as above -- all the limitations are of course open research questions, but it doesn't necessarily mean that scaling was "wrong". (The amount of money does seem crazy though, and if it screws up the US economy, I wouldn't be that surprised) Deep learning thus far is not sufficiently transparent - absolutely, the scaling has greatly outpaced understanding/interpretability Deep learning thus far has not been well integrated with prior knowledge - also seems like a valuable research direction Deep learning thus far cannot inherently distinguish causation from correlation - ditto Deep learning presumes a largely stable world, in ways that may be problematic - he uses the example of Google Flu Trends ... yes, deep learning cannot predict the future better than humans. That is a key point in the book "AI Snake Oil". I think this relates to the point about generalization -- deep learning is better at regurgitating and remixing the past, rather than generalizing and understanding the future. Lots of people are saying otherwise, and then when you call them out on their predictions from 2 years ago, they have curiously short memories. Deep learning thus far works well as an approximation, but its answers often cannot be fully trusted - absolutely, this is the main limitation. You have to verify its answers, and this can be very costly. Deep learning is only useful when verifying say 5 solutions is significantly cheaper than coming up with one yourself. Deep learning thus far is difficult to engineer with - this is still true, e.g. deep learning failed to solve self-driving ~10 years ago --- So Marcus is not wrong, and has nothing to apologize for. The scaling enthusiasts were not exactly wrong either, and we'll see what happens to their companies. It does seem similar to be dot com bubble - when the dust cleared, real value was created. But you can also see that the marketing was very self-serving. Stuff like "AGI 2027" will come off poorly -- it's an attempt by people with little power to curry favor with powerful people. They are serving as the marketing arm, and oddly not realizing it. "AI will write all the code" will also come off poorly. Or at least we will realize that software creation != writing code, and software creation is the valuable activity