Would be great to see some benchmarks on how loss changes across this very large context. It’s been technically possible to do 1mln+ token context for some time

Would be great to see some benchmarks on how loss changes across this very large context. It’s been technically possible to do 1mln+ token context for some time with performance deterioration so it would be interesting to see how this compares to those efforts