6 ms·
Yes it is, but I can imagine that they want to start out a bit smaller to see how well things scale, and/or did not yet have the time to work on optimizing for
by stingraycharles 5mo ago
Yes it is, but I can imagine that they want to start out a bit smaller to see how well things scale, and/or did not yet have the time to work on optimizing for the large context windows.
- droidjj 5mo agoI struggle to get quality results from the frontier models at contexts > 256k anyway.
- stingraycharles 5mo agoYup, same experience, it’s because the attention basically has exponential complexity. So at large context windows, they need to compress the attention (eg group multiple tokens together), when then leads to loss in accuracy. It’s almost always better to keep your context windows small.