4 ms·
The GML-4.5 model utterly fails at creating ASCII art or factorizing numbers. It can "write" Space Invaders because there are literally thousands of open source
by bgwalter 1y ago
The GML-4.5 model utterly fails at creating ASCII art or factorizing numbers. It can "write" Space Invaders because there are literally thousands of open source projects out there.
This is another example of LLMs being dumb copiers that do understand human prompts.
But there is one positive side to this: If this photocopying business can be run locally, the stocks of OpenAI etc. should got to zero.
- simonw 1y agoWhy would you use an LLM to factorize numbers?
- bgwalter 1y agoBecause we are told that they can solve IMO problems. Yet they fail at basic math problems, not only at factorization but also when probing them with relatively basic symbolic math that would not require the invocation of an external program. Also, you know it they fail they could say so instead of giving a hallucinated answer. First the models lie and say that a 20 digit number takes vast amounts of computing. Then, if pointed to a factorization program they pretend to execute it and lie about the output. There is no intelligence or flexibility apart from stealing other people's open source code.
- simonw 1y agoThat's why the IMO results were so notable: that was one of those moments where new models were demonstrated doing something that they had previously been unable to do.
- ducktective 1y agoI can't fathom why more people aren't talking about the IMO story. Apparently the model they used is not just an LLM but some RL are involved too. If a model wins gold at IMO, is it still merely a "statistical parrot"?
- sejje 1y agoStochastic parrot is the term. I don't think it's ever been accurate.
- bgwalter 1y agoThe results were private and the methodology was not revealed. Even Tao, who was bullish on "AI", is starting to question the process.
- simonw 1y agoThe same thing has also been achieved by a Google DeepMind team and at least one group of independent researchers using publicly available models and careful promoting tricks.