3 ms·
The reasoning in the article is interesting, but this struck me as a weird example to choose: > “The real question is how can we predict when a new LLM will ac
by gyomu 1y ago
The reasoning in the article is interesting, but this struck me as a weird example to choose:
> “The real question is how can we predict when a new LLM will achieve some new capability X. For example, X = “Write a short story that resonates with the social mood of the present time and is a runaway hit”
Framing a capability as something that is objectively measurable (“able to perform math on the 12th grade level”, “able to write a coherent, novel text without spelling/grammar mistakes”) makes sense within the context of what the author is trying to demonstrate.
But the social proof aspect (“is a runaway hit”) feels orthogonal to it? Things can be runaway hits for social factors independently of the capability they actually represent.
- Retric 1y agoIt’s not about being “a runaway hit” as an objective measurement it’s about the things an LLM would need to achieve before that was possible. At first AI scores on existing tests seemed like a useful metric. However, tests designed for humans make specific assumptions that don’t apply to these systems making such tests useless. AI is very good at gaming metrics so it’s difficult to list some criteria where achieving it is meaningful. A hypothetical coherent novel without spelling/grammar mistakes could in effect be a copy of some existing work that shows up in its corpus, however a hit requires more than a reskinned story.
- chii 1y ago> however a hit requires more than a reskinned story. demonstrably false with a lot of hits in the past that is a reskinned story of existing stories!
- Retric 1y agoWhile not a technical term of art, copyright applies to a reskinned story. “the series is not available in English translation, because of the first book having been judged a breach of copyright.” https://en.wikipedia.org/wiki/Tanya_Grotter https://en.wikipedia.org/wiki/Tanya_Grotter There’s plenty of room to take inspiration and go in another direction aka Pride and Prejudice and Zombies.
- creer 1y agoThat it seems hard (impossible) or not clear intuitively how to go about it, to us humans, is what makes the question interesting. In a way. The other questions are interesting but a different class of interesting. At any rate, both good for this question. Either way this becomes "what would we need to estimate this emergence threshold?".