4 ms·

What's the reason for measuring the model size in context window length and not GB? Also, are these small models OSS? Easier self hosting seems to be the main

• simion314 2y ago
  >What's the reason for measuring the model size in context window length and not GB? there are 2 different things. The context window is how many tokens ii's context can contain, so on a big model you could put in the context a few books and articles and then start your questions, on a small context model you can start a conversation and after a short time it will start forgetting eh first prompts. Big context will use more memory and will cost on performance but imagine you could give it your entire code project and then you can ask it questions, so often I know there is some functions already there that does soemthing but I can't remember the name.
• kaoD 2y ago
  I suspect you might be confusing the numbers: 12B (which is the very first number they give) is not context length, it's parameter count. The reason to use parameter count is because final size in GB depends on quantization. A 12B model at 8 bit parameter width would be 12Gbytes (plus some % overhead), while at 16 bit would be 24Gbytes. Context length here is 128k which is orthogonal to model size. You can notice the specify both parameters and context size because you need both to characterize an LLM. It's also interesting to know what parameter width it was trained on because you cannot get more information by "quantizing wider" -- it only makes sense to quantize into a narrower parameter width to save space.
  • k__ 2y ago
    Ah, yes. Thanks, I confused those numbers!
• yjftsjthsd-h 2y ago
  > Also, are these small models OSS? From the very first paragraph on the page: > released under the Apache 2.0 license.