3 ms·

Start front loading the models with 5k, 10k, 50k, 100k tokens of messy quasi related context, and then run the benchmarks. These models are ridiculously powerf

Start front loading the models with 5k, 10k, 50k, 100k tokens of messy quasi related context, and then run the benchmarks. These models are ridiculously powerful with a blank slate. It's when they get loaded down with all the necessary (and inevitably unnecessary) context to complete the task that they really start to crumble and fold.

jballanc 7mo ago
We need benchmarks that can distinguish between continuous learning and long-context extrapolation.
- vrighter 7mo ago
  oh that's easy: continuous learning is not something current architectures can do. So the benchmark for that can be done mentally