3 ms·

>Data mining doesn't mean the model is instantly updated I'm not expecting instant. Even next week it won't be there. It's like how AI never learned to count h

by panny 1y ago

>Data mining doesn't mean the model is instantly updated I'm not expecting instant. Even next week it won't be there. It's like how AI never learned to count how many times the letter r appears in strawberry. Like sure, now if you ask brave, it will tell you three, but that is only because that question went viral. It didn't "learn" anything, it was just hard coded for that particular answer. Ask it how many times the letter l appears in smallville and it will get it wrong again.

qwertytyyuu 1y ago
every week is still way to expensive to do at scale, at best they'll update training data with each model iteration.
simgt 1y ago
I didn't think for a second you could be right, so I tried with Claude. L in smallville was correct, then it suggests it'd have gotten l in parallel wrong by answering 3 instead of 2 (buts gets it right in a new chat). Then it suggests it'd get n in millennium wrong by giving the right answer, and gets it wrong in a new chat. https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 Thanks, that was enlightening.
t0md4n 1y ago
It wouldn’t be instant, next week or even next month. Pre-training doesn’t happen that frequently and varies between each model provider. As for the strawberry test, this is a tokenization issue that is fundamental to LLM’s, however, most models can now solve this type of question using thinking/code/tools to count the letters. https://imgur.com/a/NqIJEx6 https://imgur.com/a/NqIJEx6
ordersofmag 1y ago
LLM aren't retrained and released on a weekly time-scale. The data mining may only be reflected in the training of the next generation of the model.
Etheryte 1y ago
Both OpenAI and Claude average roughly one flagship release a year, and these are some of the best funded companies in the space. The bigger your model, the more expensive it is to train, so you want to do it as rarely as reasonably possible. Every other company will either work with smaller models and/or train even more rarely, aside from fine-tunes and customizations they put on top.

by panny 1y ago

qwertytyyuu 1y ago

every week is still way to expensive to do at scale, at best they'll update training data with each model iteration.

simgt 1y ago

I didn't think for a second you could be right, so I tried with Claude. L in smallville was correct, then it suggests it'd have gotten l in parallel wrong by answering 3 instead of 2 (buts gets it right in a new chat). Then it suggests it'd get n in millennium wrong by giving the right answer, and gets it wrong in a new chat. https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 Thanks, that was enlightening.

t0md4n 1y ago

It wouldn’t be instant, next week or even next month. Pre-training doesn’t happen that frequently and varies between each model provider. As for the strawberry test, this is a tokenization issue that is fundamental to LLM’s, however, most models can now solve this type of question using thinking/code/tools to count the letters. https://imgur.com/a/NqIJEx6 https://imgur.com/a/NqIJEx6

ordersofmag 1y ago

LLM aren't retrained and released on a weekly time-scale. The data mining may only be reflected in the training of the next generation of the model.

Etheryte 1y ago

Both OpenAI and Claude average roughly one flagship release a year, and these are some of the best funded companies in the space. The bigger your model, the more expensive it is to train, so you want to do it as rarely as reasonably possible. Every other company will either work with smaller models and/or train even more rarely, aside from fine-tunes and customizations they put on top.