4 ms·

Is this real input context or is it some vectordb in the background type trickery?

• HarHarVeryFunny 4y ago
  Pretty sure it's not "real" (model) context width. Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a context width of 65k, but then give a bit more detail to say they are using ALiBi - a type of positional encoding that allows longer contexts at inference time than training (i.e beyond the real context width of the model). For these types of "extended context" models to actually reason over inputs longer than the native context width of the model, I assume that there is indeed some sort of vector DB trickery - maybe paging thru the input to generate vector DB content, then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ? Maybe someone from Anthropic or MosaicML could throw us a bone and give a bit more detail of how these are working ! https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b https://arxiv.org/abs/2005.11401 https://arxiv.org/abs/2005.11401
  • sanxiyn 4y ago
    There is no trickery going on for MPT. The model is open source. MPT-7B-StoryWriter-65k+ was trained on books with 65k context length so there is nothing not "real" about it. The point of ALiBi is that you can reuse training you did with short context for long context to save compute, but of course you also just train with long context. Who knows about Anthropic.
• jxmorris12 4y ago
  appears to be real input context