3 ms·
I'd be interested to know what was going on with it during the public test as there were numerous reports of it improving considerably at tasks it was asked to
by esskay 2mo ago
I'd be interested to know what was going on with it during the public test as there were numerous reports of it improving considerably at tasks it was asked to do early on in the test compared to later in it.
- rfoo 2mo agolol don't shout out the obvious
- daveyoung 2mo agoTwo potentials from my pov: 1. Just variance in pass@K. If you prompt any model multiple times you'll see a large variance. N=1, but I find chinese open source models have a higher variance than higher-RL'd models like fable/opus. 2. They legitimately shipped a new RL checkpoint over the 7 days, which I find hard to believe. I am leaning towards 1.
- re-thc 2mo ago2. There was a new checkpoint. Official.
- daveyoung 2mo agodo you have reference to where it was said?
- zarzavat 2mo ago3. Deployment problems unrelated to the weights causing degraded performance
- swiftcoder 2mo agoFor sure the version accessible from OpenCode had a massive timeout problem the first day or so, which seemed to heavily degrade its task completion rate
- dannyw 2mo agoOr 3, they find some bug/regression in their pipeline; maybe they didn't quant parts of a model properly, maybe their inference engine had a bug, maybe some pinned MoE expert wasn't pinned, etc... That's very plausible to have, identify, and fix in a day; especially when you get community feedback in the wild.
- utilize1808 2mo agoIt's logical to serve the best version (quant) of the model at the beginning so that users keep testing it. It is also reasonable to think that the developer of the model tried to test various quant levels by gradually degrading the model's capabilities.
- brookst 2mo agoI mean that’s imaginative but not sure there’s any evidence at all for it, and it’s the opposite of what the comment you replied to observed.