Version 4 also now supports 32k tokens, good luck handling that on even awesome gaming local dev rig machines, although perhaps with linformer ideas, block-wise

Version 4 also now supports 32k tokens, good luck handling that on even awesome gaming local dev rig machines, although perhaps with linformer ideas, block-wise algorithms to handle larger than GPU memory, universal memory / RDMA it’s entirely doable. I got 50,000 atoms simulation back in 2018 on 11gb vram, at 32bit floats, the software stack has come a long way and now we have the 24gb 4090 with bfloat16, and vector DBs, and the infinite-context transformer paper just came out, so models all ought to be retrained on that if the method is truly superior anyway, not sure how atoms translate to pages of text but it’s almost surely possible to make a pretty useful LLM. Although, OpenAI has a massive moat named “data”