5 ms·
Gary Marcus discusses AI's technical problems
- m3kw9 2y agoLmao this guy keeps moving the sticks as the AI gets more powerful by the month
- itkovian_ 2y ago"That is a phrase that I coined in a 2022 essay called “Deep Learning is Hitting a Wall,” which was about why scaling wouldn’t get us to AGI. And when I coined it, everybody dismissed me and said, “No, we’re not reaching diminishing returns. We have these scaling laws. We’ll just get more data.” How can anyone think he's arguing in good faith at this point. That essay was published after gpt3 prior to gpt 4 - and he's claiming it was correct!
- abc-1 2y agoSo his timing was slightly off. I don’t know why people expected LLMs to improve exponentially. Your iPhone now doesn’t look much different than the one 10 years ago. GPT-3 or arguably GPT-4 was the first iPhone moment, everything else will be gradual improvements unless fundamental discoveries are found, but those happen seemingly randomly.
- pj_mukh 2y agoThe difference between a doomsday conspiracy theorist and a physicist surmising the heat death of the universe is...just timing.
- anonylizard 2y agoIf one compares O3-mini's coding abilities to the original GPT-4. It is as large as GPT-3 to GPT-4 gap GPT-3: Useful as autocomplete. Still error prone, but vastly better than any pre-AI autocomplete GPT-4: Already capable of independently coding up simple functions based on natural language. O3-mini: Can code in say top 5% of codeforces. There's a 2 years gap between each of them. More over, intelligence has a superexponential return, 90IQ->100IQ < 100IQ->110IQ in terms of returns.
- ben_w 2y ago> More over, intelligence has a superexponential return, 90IQ->100IQ < 100IQ->110IQ in terms of returns That's the second time I've seen the claim that linear increases in intelligence have "superexponential" results, first time was Altman's blog. But I've not seen any justification for this. (As you specifically say IQ, note that an IQ is defined as a mapping of standard deviations rather than a mapping of absolute skill, the normal mapping is 15 points being 1σ).
- mquander 2y agoIn what sense are the bleeding edge models incremental improvements over GPT-3 (read his examples of GPT-3 output and imagine any of the top models today producing them!), GPT-3.5, or GPT-4? Look at any benchmark or use it yourself. It's night and day. Gary Marcus didn't make a lot of specific criticisms or concrete predictions in his essay [0], but some of his criticisms of GPT-3 were: - "For all its fluency, GPT-3 can neither integrate information from basic web searches nor reason about the most basic everyday phenomena." - "Researchers at DeepMind and elsewhere have been trying desperately to patch the toxic language and misinformation problems, but have thus far come up dry." - "Deep learning on its own continues to struggle even in domains as orderly as arithmetic." Are these not all dramatically improved, no matter how you measure them, in the past three years? [0] https://nautil.us/deep-learning-is-hitting-a-wall-238440/ https://nautil.us/deep-learning-is-hitting-a-wall-238440/
- suddenlybananas 2y agoWell it's true that all of the most recent advances come from changes the architecture to do to inference scaling instead of model scaling. Scaling laws as people talked about in them in 2022 (that you take a base LLM and make it bigger) are dead.
- menaerus 2y agoI think you want both. To scale the model, e.g. train it with more and more data, you also need to scale your inference step. Otherwise, it just takes too long and it's too costly, no?
- sytelus 2y agoI lost faith in Marcus after just a few interactions. He is indeed what one would refer to as "crackpot" in academia. The most glaring thing was that he is technically extremely shallow and don't have a clue about most of the details. I also got impression that is enormously enamored with having attention and recognition at any cost. Depending on weather, he will change directions and views, basically just do anything it took to get that attention no matter how ridiculous he looks doing that. While writing this, it occured to me that he would get even goose bumps at reading this comment because it, after all, I am giving him attention.
- nonrandomstring 2y agoI thought "Attention Is All You Need"
- baobabKoodaa 2y ago> Depending on weather, he will change directions and views My impression is the opposite: I would describe Gary Marcus as having all his opinions perfectly aligned to a singular viewpoint at all times regardless of weather (or evidence).
- sytelus 2y agoAt different points he has claimed (1) current models are not intelligent at all and can never be intelligent like us, (2) also we need to ban current models because they will outsmart us and take over the world. Depending on how he can get interview or get seat at the table, he may chose exact opposite of positions.
- baobabKoodaa 2y agoCan you show an example of 2)? I've missed that.
- deleted 2y ago[deleted]
- petters 2y agoI don't think he is always arguing in good faith, unfortunately.
- ripped_britches 2y agoThis poor guy
- pinoy420 2y agoWho is it
- ripped_britches 2y ago#1 AI progress denier, got stuck in a narrative he can’t let go of
- _giorgio_ 2y agoJust an ignorant and non technical guy trying to stay relevant. He has an infinite quantity of time and somehow always finds someone to interview him about the fact that AIs don't work and have been hitting a wall since the last ten years. He invents some metrics that nobody has agreed on, and says that they're below that.
- baobabKoodaa 2y agoCan we all collectively just decide to stop posting Gary Marcus ragebait? Please.
- suddenlybananas 2y agoIt's remarkable how uncomfortable AI guys get about criticism. Why is it a bad thing to criticise the models? The models still have enormous gaps in their capacities (e.g. river crossing questions where you make them simple instead of difficult).