4 ms·

Just want to confirm: so this is the first locally runnable model with a context length of greater than 128K and it’s gone straight to 1M, correct?

• terhechte 2y ago
  Yes. It requires a lot of ram, and even on a M4 with a lot of ram, if you give it 1mio tokens the prompt processing alone (that is, before you get the first response token) will probably take ~30min or more. However I'm looking forward to check if indeed I can give it a whole codebase and ask questions about it.
  • deleted 2y ago
    [deleted]
  • marci 2y ago
    You might want to try caching to a file with mlx. https://github.com/ml-explore/mlx-examples/pull/956 https://github.com/ml-explore/mlx-examples/pull/956 edit: here's a quick example for qwen2.5-1M from a mlx dev https://x.com/awnihannun/status/1883611098081099914 https://x.com/awnihannun/status/1883611098081099914
    • terhechte 2y ago
      That's cool, than you, but does MLX support the Qwen 1M context yet?
      • marci 2y ago
        According to the tweet, not the full context yet.
• segmondy 2y ago
  No, this is not the first local model with a context length of greater than 128k, there have been such models, for example the following https://huggingface.co/ai21labs/AI21-Jamba-1.5-Mini https://huggingface.co/ai21labs/AI21-Jamba-1.5-Mini 256k https://huggingface.co/THUDM/glm-4-9b-chat-1m https://huggingface.co/THUDM/glm-4-9b-chat-1m 1M and many other's that supposedly extended traditional models via finetune/rope scaling
  • mmaunder 2y ago
    Thanks.