4 ms·
There is no trickery going on for MPT. The model is open source. MPT-7B-StoryWriter-65k+ was trained on books with 65k context length so there is nothing not "r
by sanxiyn 3y ago
There is no trickery going on for MPT. The model is open source. MPT-7B-StoryWriter-65k+ was trained on books with 65k context length so there is nothing not "real" about it. The point of ALiBi is that you can reuse training you did with short context for long context to save compute, but of course you also just train with long context.
Who knows about Anthropic.