4 ms·
"Investors are putting pressure, change the version number now!!!"
by system2 10mo ago
"Investors are putting pressure, change the version number now!!!"
- exe34 10mo agoI'm quite sad about the S-curve hitting us hard in the transformers. For a short period, we had the excitement of "ooh if GPT-3.5 is so good, GPT-4 is going to be amazing! ooh GPT-4 has sparks of AGI!" But now we're back to version inflation for inconsequential gains.
- verdverm 10mo ago2025 is the year most Big AI released their first real thinking models Now we can create new samples and evals for more complex tasks to train up the next gen, more planning, decomp, context, agentic oriented OpenAI has largely fumbled their early lead, exciting stuff is happening elsewhere
- ToValueFunfetti 10mo agoTake this all with a grain of salt as it's hearsay: From what I understand, nobody has done any real scaling since the GPT-4 era. 4.5 was a bit larger than 4, but not as much as the orders of magnitude difference between 3 and 4, and 5 is smaller than 4.5. Google and Anthropic haven't gone substantially bigger than GPT-4 either. Improvements since 4 are almost entirely from reasoning and RL. In 2026 or 2027, we should see a model that uses the current datacenter buildout and actually scales up.
- snovv_crash 10mo agoDatacenter capacity is being snapped up for inference too though.
- Leynos 10mo ago4.5 is widely believed to be an order of magnitude larger than GPT-4, as reflected in the API inference cost. The problem is the quantity of parameters you can fit in the memory of one GPU. Pretty much every large GPT model from 4 onwards has been mixture of experts, but for a 10 trillion parameter scale model, you'd be talking a lot of experts and a lot of inter-GPU communication. With FP4 in the Blackwell GPUs, it should become much more practical to run a model of that size at the deployment roll-out of GPT-5.x. We're just going to have to wait for the GBx00 systems to be physically deployed at scale.
- JanSt 10mo agoI don't feel the S-curve at all yet. Still an exponential for me
- exe34 10mo agoWith a very long doubling time?
- gessha 10mo agoBecause it will take thousands of underpaid researchers random searching through solution space to get to the next improvement, not 2-3 companies pressed to monetize and enshittify their product before money runs out. That and winning more hardware lotteries.
- astrange 10mo agoUnderpaid? OpenAI!? It's pretty good I think. https://www.levels.fyi/companies/openai/salaries/software-engineer/title/research-scientist?country=254 https://www.levels.fyi/companies/openai/salaries/software-en...
- gessha 10mo agoI’m talking about grad students, not OpenAI researchers.