Yes definitely, you have a valid concern. For long contexts this is a key consideration along with what self attention optimizations the model chooses to imple

Yes definitely, you have a valid concern. For long contexts this is a key consideration along with what self attention optimizations the model chooses to implement. They don’t make this public, but we can infer they can’t be using full self attention pairs at 1,000,000 tokens because it scales quadratically and would take Terabytes of RAM. There are different approaches like sparse attention, and the only way to really know how well their choices work is to test it.