3 ms·

i think there is a very very real limit to LLM technology that hasn’t been publicly solved yet. fundamentally, how do you get it to do something new? openAI’s m

i think there is a very very real limit to LLM technology that hasn’t been publicly solved yet. fundamentally, how do you get it to do something new? openAI’s moat is their training data, but again, all machine learning techniques fundamentally don’t get a model to produce something novel. they quite literally have to do something that’s been done before, maybe spelled a little differently. until that’s solved, i feel LLM’s will always have a peak below a skilled human.

atleastoptimal 2y ago
That depends on what counts as "new". It's very easy to humans and LLM to produce something new. Just ask them "make up a new word and tell me its definition" or "write a story about (random thing a) and (random thing b). What I'm sure you're referring to is "new" and "useful", which is the standard by which we judge the utility and force of innovation. Very few humans could even do this. The problem is partially due to many of the "low-hanging fruit" innovations having already been discovered, along with the fact that creating something of value requires an intersecting intuition over many domains, modalities, considerations, etc. It is a test of general intelligence, and AI has not gotten there yet, but there's no reason to think that this is a fundamental limit to LLM's.