3 ms·

I'd love to see someone go for another few epochs in the future. Two of the benchmarks got a significant jump almost at the end of training. I wonder if there's

I'd love to see someone go for another few epochs in the future. Two of the benchmarks got a significant jump almost at the end of training. I wonder if there's a chance for more of that - looks like an interesting effect on its own.

smpanaro 3y ago
The jump was due to them fixing a bug. There’s a footnote about it on the bottom of page 5. In the Discord, they mentioned a TinyLLaMa v2, presumably that would have this bug (and another bug, footnote page 4) fixed.