> I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Scaling can happen in two dimensions - model si

> I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Scaling can happen in two dimensions - model size and dataset size. What counts is the product of n_examples x n_parameters. That's why we have the super-Chinchilla laws, where n_examples >> 20*n_parameters. Scale the data, keep the model lean. Not to mention dataset quality - if you got clean diverse data you need less of it. Another important trend is using LLMs to generate training sets. Example: ConstitutionalAI (pure RLAIF), TinyStories (scaling down LLMs), Alpaca (borrowed RLHF), AlpacaFarm - a recent paper promising fine-tuned models for $200 cost in 24h.