2 ms·

Thanks - still not clear what they did really. Some inference time hacks?

• FergusArgyll 1y ago
  That would imply the model always had a 1m token context but they limited it in the api and app? That's strange because they can just charge more for every token past 250k (like google does, I believe). But if not shouldn't it have to be completely retrained model? it's clearly not that - good question!
• Aeolun 1y ago
  They already had 0.5M context window on the enteprise version.
• otabdeveloper4 1y ago
  Most likely still 32k tokens under the hood, but with some context slicing/averaging hacks to make inference not error out on infinite input. (That's what I do locally with llama.cpp)