Not sure exactly what else the MosaicML team did to release their MPT-7B-StoryTeller-65k+ with ALiBi but I did hear interviews where they say it can be extended

Not sure exactly what else the MosaicML team did to release their MPT-7B-StoryTeller-65k+ with ALiBi but I did hear interviews where they say it can be extended to an unknown extent by adding compute and I don't think they were talking 100:1 This seems like part of the conversation where advances in software are expected to deliver bigger gains than hardware for a while. >At inference time, thanks to ALiBi, MPT-7B-StoryWriter-65k+ can extrapolate even beyond 65k tokens. We demonstrate generations as long as 84k tokens on a single node of 8 A100-80GB https://huggingface.co/mosaicml/mpt-7b-storywriter https://huggingface.co/mosaicml/mpt-7b-storywriter Edit: here is a snip of the interview - https://share.snipd.com/snip/93d96e73-9841-4361-bd82-ae086e165742 https://share.snipd.com/snip/93d96e73-9841-4361-bd82-ae086e1...