45 ms·
It’s not just OpenAI. I think AI winter is coming much sooner than expected. Throwing more compute at the current solution is not yielding better results it see
by msoad 2y ago
It’s not just OpenAI. I think AI winter is coming much sooner than expected. Throwing more compute at the current solution is not yielding better results it seems. Reasoning will not be solved by a bigger LLM
- Davidzheng 2y agoTo quote Geoff Hinton "AI winter !!!!????"
- CamperBob2 2y agoWhat did he mean by that?
- bloppe 2y agohttps://en.m.wikipedia.org/wiki/AI_winter https://en.m.wikipedia.org/wiki/AI_winter It's been a pattern in AI research since the 70s. Sure, the current boom is unprecedented, but that doesn't mean there won't be a relative bust. AI winter doesn't mean chatgpt will disappear. It just means research funding may get significantly scaled back of the hundred billion dollar investments of today don't generate trillion dollar returns
- CamperBob2 2y agoI know. What did Hinton mean by his comment, though?
- tim333 2y agoAfter spending more time researching than I really should have done I found the quote. https://x.com/pmddomingos/status/1728628326968021100 https://x.com/pmddomingos/status/1728628326968021100 He was expressing doubt that there is an AI winter near. He's more of the: >Geoffrey Hinton, dubbed the 'Godfather of AI,' warns technology will be smarter than humans in five years" school of thought. https://www.dailymail.co.uk/sciencetech/article-12610845/geoffrey-hinton-godfather-ai-warns-smarter-humans.html https://www.dailymail.co.uk/sciencetech/article-12610845/geo...
- faeriechangling 2y agoI’ve sure seen better results from throwing more compute at the problem and we’re going to have a lot more compute in 20 years than we have now.
- deleted 2y ago[deleted]
- prng2021 2y agoHow has throwing compute at the problem not yielded better results? Are you denying that generative AI has been improving by leaps and bounds throughout the last few years?
- famouswaffles 2y agoSome people are convinced that because we don't have a next gen LLM model (i.e GPT-5) 14 months after 4 and that the best 3rd party models are 'only' GPT-4 level, there must be a plateau or something. Nevermind the 33 month gap between 3 and 4 or the 18 month gap between GPT-3 and the first 3rd party >= GPT-3 level model (Gopher).
- coffeebeqn 2y agoYeah this is like saying after the Intel Pentium came out that welp no one has built an exponentially better CPU in the last year so we’ve peaked as a CPU building species. Even AMDs latest only brought it up up to the same level... CPU winter 1994
- pedalpete 2y agoI actually think it's worse than that. HTML5 came out in 2008, so nobody could possibly come up with a new internet business until we progress to HTML6! So without a new release of the language, all development stops!
- doubloon 2y agodisagree. AI has not even begun to penetrate its potential in corporate America. you could enhance so much middleware stuff with a good unstructured data organization AI and slap voice recog on top of it, you have massively multiplied the effectiveness of a whole bunch of mid level people from operations staff to sales reps, who are very good at their domain but continually stymied by computer syntax and lack of programmers to make the computer do stuff.
- renegade-otter 2y agoMost businesses do not even have their data game on properly. AI is an iteration of Big Data, and BOTH need proper source material, or the value will be zero AT BEST. https://hbr.org/2013/12/you-may-not-need-big-data-after-all https://hbr.org/2013/12/you-may-not-need-big-data-after-all
- megadal 2y agoThe use case you described can be achieved with a subset of NLP, and has basically already been possible for years. Named-Entity Recognition (NER) and Text Classification will allow you to figure out what kind of text you're looking at and extract structured data. LLMs are not good at this because they're not specialized for it, but you can build a specialized NER model to extract custom entities from unstructured data today. That said, I don't really think this is some yet untapped potential of AI so much as an area of ML that just hasn't been applied enough. ETA: also, in general I think AI is going in the direction of basically just having an LLM route tasks to more specialized ML models (for corporate tasks at least). That's what Google's Vertex AI agents sort of do (and I am guessing the GPT 4 agents as well).
- KoolKat23 2y agoBut this is the thing, that requires expertise and infrastructure. "AGI" LLM's can handle all the nuance, quite simply, they don't need specialist infrastructure or specialist programming, way cheaper, upfront cost. Way easier to scale. Individual employees can ask it for specific things that make their lives easier and it'll give it to them/or do it. No need to ask your manager, motivate for funding and hire an engineer/purchase new software/equipment.
- yxgao 2y agoI don’t think it’s going to be a “winter”, but there are definitely some bubbles to burst. Especially when LLMs become half-assed products and the general public’s heightened expectations are not met.
- megadal 2y agoI think the biggest bubble that needs to burst is probably the whole "AGI" thing. The definition of it isn't clear but from what I gather it's basically an aggregate of emergent capabilities that work together to produce a singularity. Maybe with enough resources it's possible but I highly doubt it'll be economically feasible given how much has gone into it so far and how far we really are away from something like that with current models.
- adammarples 2y agoIn my opinion you could make an LLM 100x bigger and it would only get better at generating the next token in a sentence. And everyone knows that the best sentences are not constructed by the most intelligent people with the most accurate world model, but by the people who are best at constructing sentences. It's a dead end in terms of real intelligence and reasoning imo.
- KoolKat23 2y agoPeople that make the best sentences don't necessarily make the world go round. In most scenario's, a barely adequate sentence is enough to keep the world turning..
- notarobot123 2y agoAGI, in the sense you mention, is an imagined/hoped-for supreme-power that will save/destroy us all (or maybe just the "worthy"/"unworthy" ones). In an age of such hopelessness about the future, this looks a lot like an emotional crutch wrapped in the veil of rationality - just the thing an anxious materialist needs to make sense of the world. Like many cults and religions it mistakes the plausible for possible and possible for probable. The problem with religious beliefs like these is that they don't just disappear with evidence or sufficient reasoning. I don't think that particular bubble is bursting anytime soon.
- mvkel 2y agoOn what evidence are you making this statement? We don't even know how existing LLMs work, not really, yet we're done? All signs are pointing to there still being no upper limit on results. Not yet. Just because it's not on the leaderboard yet, doesn't mean we've heard the bell.
- bozey07 2y agoOf course "we" know how LLMs work. Nobody would be able to make one, otherwise.
- mvkel 2y agoJust because you can make something doesn't mean you know why it's made. There are thousands of people around the world trying to reverse engineer what is going on in the billions or trillions of parameters in an LLM. It's a field called "Mechanistic Interpretability." The people who do the work jokingly call it "cursed" because it is so difficult and they have made so little progress so far. Literally nobody can predict before they are released what capabilities new models will have in them. And then, months after a model is released, people discover new abilities in it, such as decent chess playing. They are black boxes.
- irthomasthomas 2y agoI predict that this is largely an illusion staged by the lack of publishing of the datasets and training regime used. Also an artefact of how evals have been done on a pass fail basis. So that an LLM that gets 90% of a question right is just as much a failure as one that gets 0% of the question. So that skills appear to emerge suddenly and surprisingly only due to the flawed way that we are forced to study them. Consider the training regime, and partial success towards a goal, and emergence is far less prevalent. There was a paper on that recently, I'll see if I can find.
- mvkel 2y agoUntil <5 years ago, AI was almost entirely a purely academic field, theoretical at that. Those same academics admit themselves that they're surprised at how well LLMs do considering how simple(?) rudimentary(?) the logic underneath is. I don't quite understand what you're saying. That these academics were being lazy by not properly investigating/publishing their findings? That doesn't seem right.
- vivzkestrel 2y agobecause the amount of data you can train any model on is limited. After training your AI model on the 1st quadrillion images, trillion videos, where do you find the next quadrillion? This is going to be the limitation on top of law of diminishing returns
- echelon 2y ago> After training your AI model on the 1st quadrillion images, trillion videos, As far as images and video are concerned, we're done. Frame generation is almost perfect, and we don't really need any more training data. Now it's time to build product and enhance how the models work. LLMs, though? That field appears to have hit a wall for now. AI for media is going to be a rocket ship. AI for knowledge and text and reasoning will take longer. People will recognize this soon.
- vineyardmike 2y ago> AI for media is going to be a rocket ship. Is there actually enough data to train an AI on? Photos seem to be a success, like you say, but everything else? I’ve heard many people make claims about where AI content will be most useful. A persistent theme is an AI made VR world, customized video games, and endless AI generated TikTok videos. I genuinely question if the training data exists for these use cases. Photos are cheap and easy, but quality annotated 3D models? Short form videos? What about long form video? Are we really there (assuming inference was cheap)?
- echelon 2y ago> Short form videos? What about long form video? These are being solved as we speak. I'm working on this problem directly and the level of control and consistency achievable is incredible. Video is just a special case of images. Take a look at the ComfyUI space and the authors of plugins and papers. > quality annotated 3D models The research is progressing at a fast pace. We can get good surface topologies, textures, and there are teams working on everything from rigging to animation.
- 2y ago
- tonyhart7 2y agothats why microsoft want to build supercomputer to do it, You know that microsoft is backing Open AI right. They dont want lose to Apple,Google and Meta
- hackerlight 2y ago4o uses less compute and performs better.
- deleted 2y ago[deleted]
- yawpitch 2y agoThe main question is what will be the Venn overlap between AI Winter 2.0 and Nuclear Winter 1.0?
- cedws 2y agoIt can't come soon enough. Expectations and hype have reached stratospheric levels and are due for a hard correction. Every company is jamming "AI" into places it isn't needed to juice their share price and please the clueless shareholders. Despite LLMs popping up everywhere there's limited evidence to support the claimed productivity boosts. Other than using GitHub Copilot I don't know a single person who seriously uses language models for work, whether it's by prompting directly or pressing a magic "AI" button that performs some RAG. I've half seriously considered the possibility a large portion of the hype has been manufactured in an attempt to shock stagnating economies back to life, post-COVID, post low interest rates.