7 ms·

So today is Qwen. Tomorrow a new SOTA model from Google apparently, R2 next week. We haven't hit the wall yet.

by gatienboquet 2y ago

behnamoh 2y ago
Google's announcements are mostly vaporware anyway. Btw, where is Gemini Ultra 1? how about Gemini Ultra 2?
- karmasimida 2y ago
  It is already on the LLM arena right, codename nebula? But you are right they can fuck up their releases royally.
- aoeusnth1 2y ago
  I guess they don’t do ultras anymore, but where was the announcement for it? What other announcement was vaporware?
zamadatix 2y ago
Qwen 3 is coming imminently as well https://github.com/huggingface/transformers/pull/36878 https://github.com/huggingface/transformers/pull/36878 and it feels like Llama 4 should be coming in the next month or so. That said none of the recent string of releases has done much yet to "smash a wall", they've just met the larger proprietary models where they already were. I'm hoping R2 or the like really changes that by showing ChatGPT 3->3.5 or 3.5->4 level generational jumps are still possible beyond the current state of the art, not just beyond current models of a given size.
- YetAnotherNick 2y ago
  > met the larger proprietary models where they already were This is smashing the wall. Also if you just care about breaking absolute numbers, OpenAI released 4.5 a month back which is SOTA in base model, planning to release O3 full in maybe a month, and Deepseek released new V3 which is again SOTA in many aspects.
tomdekan 2y ago
Any more info on the new Google model?
OsrsNeedsf2P 2y ago
> We haven't hit the wall yet. The models are iterative improvements, but I haven't seen night and day differences since GPT3 and 3.5
- anon373839 2y ago
  Yeah. Scaling up pretraining and huge models appears to be done. But I think we're still advancing the frontier in the other direction -- i.e., how much capability and knowledge can we cram into smaller and smaller models?
- Davidzheng 2y ago
  Tbh such a big jump from current capability would be ASI already
- YetAnotherNick 2y ago
  Because 3.5 has a new capability which is following instructions. Right now we are in 3.5 range in conversation AI and native image generation, both of which feels magical.
nwienert 2y ago
We've slid into the upper S curve though.
intalentive 2y ago
Asymptotic improvement will never hit the wall

So today is Qwen. Tomorrow a new SOTA model from Google apparently, R2 next week. We haven't hit the wall yet.

behnamoh 2y ago

Google's announcements are mostly vaporware anyway. Btw, where is Gemini Ultra 1? how about Gemini Ultra 2?

karmasimida 2y ago
It is already on the LLM arena right, codename nebula? But you are right they can fuck up their releases royally.
aoeusnth1 2y ago
I guess they don’t do ultras anymore, but where was the announcement for it? What other announcement was vaporware?

zamadatix 2y ago

Qwen 3 is coming imminently as well https://github.com/huggingface/transformers/pull/36878 https://github.com/huggingface/transformers/pull/36878 and it feels like Llama 4 should be coming in the next month or so. That said none of the recent string of releases has done much yet to "smash a wall", they've just met the larger proprietary models where they already were. I'm hoping R2 or the like really changes that by showing ChatGPT 3->3.5 or 3.5->4 level generational jumps are still possible beyond the current state of the art, not just beyond current models of a given size.

YetAnotherNick 2y ago
> met the larger proprietary models where they already were This is smashing the wall. Also if you just care about breaking absolute numbers, OpenAI released 4.5 a month back which is SOTA in base model, planning to release O3 full in maybe a month, and Deepseek released new V3 which is again SOTA in many aspects.

tomdekan 2y ago

Any more info on the new Google model?

OsrsNeedsf2P 2y ago

> We haven't hit the wall yet. The models are iterative improvements, but I haven't seen night and day differences since GPT3 and 3.5

anon373839 2y ago
Yeah. Scaling up pretraining and huge models appears to be done. But I think we're still advancing the frontier in the other direction -- i.e., how much capability and knowledge can we cram into smaller and smaller models?
Davidzheng 2y ago
Tbh such a big jump from current capability would be ASI already
YetAnotherNick 2y ago
Because 3.5 has a new capability which is following instructions. Right now we are in 3.5 range in conversation AI and native image generation, both of which feels magical.

nwienert 2y ago

We've slid into the upper S curve though.

intalentive 2y ago

Asymptotic improvement will never hit the wall