the 2048 sequence length makes it uncompetitive, especially now that we are entering ~infinite-length bots. $1M for training is peanuts for FB. There are open d

the 2048 sequence length makes it uncompetitive, especially now that we are entering ~infinite-length bots. $1M for training is peanuts for FB. There are open datasets like redPajama and alternative models are (hopefully) coming up