Pretty sure it's not "real" (model) context width. Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a conte

Pretty sure it's not "real" (model) context width. Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a context width of 65k, but then give a bit more detail to say they are using ALiBi - a type of positional encoding that allows longer contexts at inference time than training (i.e beyond the real context width of the model). For these types of "extended context" models to actually reason over inputs longer than the native context width of the model, I assume that there is indeed some sort of vector DB trickery - maybe paging thru the input to generate vector DB content, then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ? Maybe someone from Anthropic or MosaicML could throw us a bone and give a bit more detail of how these are working ! https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b https://arxiv.org/abs/2005.11401 https://arxiv.org/abs/2005.11401