4 ms·

You are assuming the issues with LLM's today will persist into the future and that they will not improve. Perhaps consider that the mass adoption of genAI by ma

You are assuming the issues with LLM's today will persist into the future and that they will not improve. Perhaps consider that the mass adoption of genAI by major tech companies isn't entirely due to an assumption that plugging in whatever is available now will be useful, but rather that their efforts are for the sake of establishing footing and infrastructure in a domain which will prove useful in the near future. They are building highways when cars can travel 25mph because they know soon they'll be able to travel 50mph. All economic activity is based on the assumption that the future is a real thing.

pokerface_86 2y ago
i think there is a very very real limit to LLM technology that hasn’t been publicly solved yet. fundamentally, how do you get it to do something new? openAI’s moat is their training data, but again, all machine learning techniques fundamentally don’t get a model to produce something novel. they quite literally have to do something that’s been done before, maybe spelled a little differently. until that’s solved, i feel LLM’s will always have a peak below a skilled human.
- atleastoptimal 2y ago
  That depends on what counts as "new". It's very easy to humans and LLM to produce something new. Just ask them "make up a new word and tell me its definition" or "write a story about (random thing a) and (random thing b). What I'm sure you're referring to is "new" and "useful", which is the standard by which we judge the utility and force of innovation. Very few humans could even do this. The problem is partially due to many of the "low-hanging fruit" innovations having already been discovered, along with the fact that creating something of value requires an intersecting intuition over many domains, modalities, considerations, etc. It is a test of general intelligence, and AI has not gotten there yet, but there's no reason to think that this is a fundamental limit to LLM's.
dangus 2y ago
I'm assuming that LLMs will hit the plateau that all emerging technology hits. Just because cars used to travel at 25 MPH and their speed increased to 75 MPH doesn't mean we are going to drive them at 300 MPH anytime in the future.
- atleastoptimal 2y ago
  The point is that the plateau could be 6 months from now or decades from now, you don't know, and can't make a judgement solely on the idea that all tech hits plateaus. Moore's law held steady for decades. Nvidia's chip improvements, the main driver behind exponential gains in AI compute per dollar, are not close to slowing down. Even if the current generation of chips is the last one forever (almost 0% chance this is the case), all the SOTA models today were trained on chips from 1-2 years ago, there's still a 2-4x gain per dollar to be made for free. I'll believe LLM's plateau when they do plateau, but judging on the rate of development in algorithmic advances, chip capacity, multi-modal integration to broaden training data, investment, etc, it seems silly to just assume it will just plateau at a level no more significant that the level we are at now.

pokerface_86 2y ago

i think there is a very very real limit to LLM technology that hasn’t been publicly solved yet. fundamentally, how do you get it to do something new? openAI’s moat is their training data, but again, all machine learning techniques fundamentally don’t get a model to produce something novel. they quite literally have to do something that’s been done before, maybe spelled a little differently. until that’s solved, i feel LLM’s will always have a peak below a skilled human.

atleastoptimal 2y ago
That depends on what counts as "new". It's very easy to humans and LLM to produce something new. Just ask them "make up a new word and tell me its definition" or "write a story about (random thing a) and (random thing b). What I'm sure you're referring to is "new" and "useful", which is the standard by which we judge the utility and force of innovation. Very few humans could even do this. The problem is partially due to many of the "low-hanging fruit" innovations having already been discovered, along with the fact that creating something of value requires an intersecting intuition over many domains, modalities, considerations, etc. It is a test of general intelligence, and AI has not gotten there yet, but there's no reason to think that this is a fundamental limit to LLM's.

dangus 2y ago

I'm assuming that LLMs will hit the plateau that all emerging technology hits. Just because cars used to travel at 25 MPH and their speed increased to 75 MPH doesn't mean we are going to drive them at 300 MPH anytime in the future.

atleastoptimal 2y ago
The point is that the plateau could be 6 months from now or decades from now, you don't know, and can't make a judgement solely on the idea that all tech hits plateaus. Moore's law held steady for decades. Nvidia's chip improvements, the main driver behind exponential gains in AI compute per dollar, are not close to slowing down. Even if the current generation of chips is the last one forever (almost 0% chance this is the case), all the SOTA models today were trained on chips from 1-2 years ago, there's still a 2-4x gain per dollar to be made for free. I'll believe LLM's plateau when they do plateau, but judging on the rate of development in algorithmic advances, chip capacity, multi-modal integration to broaden training data, investment, etc, it seems silly to just assume it will just plateau at a level no more significant that the level we are at now.