The comments here are kinda silly… the haystack test measures how well a model can natively attend to its entire context window. Of course a more elaborate pip

The comments here are kinda silly… the haystack test measures how well a model can natively attend to its entire context window. Of course a more elaborate pipeline, or a way for the model to use a shell, or whatever will easily (trivially) solve the problem. But that’s not the point, the point is a task that’s trivial to generate and exercises 10s-100s of thousands of tokens of context in a falsifiable way.