Few points: 1. Chinchilla has demonstrated models are currently unnecessarily large, and would benefit more from data scaling. 2. Models can be brought down i

Few points: 1. Chinchilla has demonstrated models are currently unnecessarily large, and would benefit more from data scaling. 2. Models can be brought down in size massively by a combination of distillation and quantization. A GPT-3 equivalent with 50B parameters quantized to fp4 is 200gig, and it could probably be distilled to half of that or less while still being functional for the vast majority of prompts. That means ~100gig memory will be a target for devices in the near future. Once large language models are the main thing people buy GPUs (and even new computers) for, architectures will be redesigned to improve gpu -> memory bandwidth and latency. I wouldn't be surprised to see GPU integrated motherboards as a future premium tier offering, we're already running into heat and space issues with add-on cards and it should be possible to build a low latency bus to a unified system memory.