5 ms·
I have no affiliation with them but here's what I think happened: 1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex
by rafaquintanilha 4mo ago
I have no affiliation with them but here's what I think happened:
1. They claim the official model is based on Qwen 397B. It's likely they didn't disclose Nex Pro at all because Nex itself is based on the same base model (not saying they shouldn't).
2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all.
3. It's important to notice they didn't advertise the model besides posting it on Reddit 2 days ago. It became viral organically, over the weekend, and during Brazil's World Cup debut (Brazilians will understand). Of course the mayor of Rio took the opportunity to capitalize over the free coverage, but that wasn't done in conjunction with the researchers.
4. I don't see why they would disclose Qwen 397B as base and mention the SwiReasoning paper but not mention Nex if all they did was to merge both models.
5. In any case, what they are claiming is easily verifiable once (if) they upload the right model.
- throwa356262 4mo agoRegarding #2 https://news.ycombinator.com/item?id=48529544 https://news.ycombinator.com/item?id=48529544
- xiphias2 4mo agoThis should be at the top: they uploaded the wrong model, they fixed it
- jwitthuhn 4mo agoThey did upload the wrong model but as of the time of writing they have not fixed it. Right now, 12 hours after they took the old one down, there is simply no model present in their huggingface repo.
- xiphias2 4mo agoI guess they will upload it later, it seems like an honest mistake to me. Anyways SwiTransformer paper looks interesting and doing a post training to optimize for it looks interesting as well.
- 00index 4mo agoLooks like they couldn't find the correct weights. https://x.com/IplanRio_rj/status/2066693494769348946 https://x.com/IplanRio_rj/status/2066693494769348946
- matheusmoreira 4mo agoI'm honestly impressed that this even happened at all. "Rio de Janeiro's homegrown LLM" is probably the last headline I ever expected to read on HN.
- cscheid 4mo agoYes! That "prefeitura do Rio" huggingface URL is definitely shocking to read to this Brazilian as well (I'm assuming you and parent also are from your usernames).
- airstrike 4mo agoWorth reminding everyone that Lua was also created in Rio, though admittedly at PUC rather than by the government. Rio has a strong engineering talent pool, along with many other major capitals in Brazil
- deleted 4mo ago[deleted]
- matheusmoreira 4mo agoBrazil does have talent. Mauro Carvalho Chehab is a Linux kernel maintainer. Elixir was created by José Valim, a brazilian. I have also created my own programming language. What Brazil doesn't have is a history of properly rewarding talent, which often causes it to migrate elsewhere. So it's definitely surprising when any sort of technological development happens in Brazil: it implies someone who stayed managed to get something done, most likely for much less than what that something is actually worth, while also being crushed by extremely high taxes that essentially doubles the cost of computer hardware.
- rbanffy 4mo ago> extremely high taxes I always find this funny. Brazilian taxes are nowhere near what I would say “high”. I pay about twice as much out of my compensation as I would pay in Brazil, and that would be as if I did zero tax optimisation back then.
- Aurornis 4mo ago> 2. The improvement would come from merging the weights PLUS on-policy distillation. The confusion is that the uploaded model didn't have the distillation at all. They merged the base model with another lab’s fine tuned model. The improvements could have come from getting some of the fine tuned weights from the other model. If they really had a better performing model that they “accidentally” forgot to upload, they could have uploaded the correct file by now.
- croes 4mo agoSeems they did https://news.ycombinator.com/item?id=48529544 https://news.ycombinator.com/item?id=48529544
- ipieter 4mo agoI only see an edit to the readme (13h ago) and removal of the weights, so the repo is now empty. I am willing to give them the benefit of the doubt, but we've seen this before: a model gets released that is supposedly state-of-the-art, yet seems to be a an other repackaged model without any training. Reflection 70B was the most similar example, all they now need is an api that rewrites "Claude" to "Rio".
- smus 4mo agoWhat do you mean World Cup debut? haven't they won 5?
- alxndresp 4mo agoThey meant their first, opening game of this current World Cup tournament
- s1artibartfast 4mo agoMy understanding is that they didnt do any distalation. Tevery weight is a 60/40 element wise average of QWEN and NEX. Is this possible if the rio contracter did thei own post-training as claimed? https://x.com/tenobrus/status/2066243352211996728/photo/1 https://x.com/tenobrus/status/2066243352211996728/photo/1
- motbus3 4mo agoIt seems to me this is clearly a mistake. They would not even have the resources for it as far as I know and I think they are not even on a position to such bold claims.
- matheusmoreira 4mo agoBrazil could easily do it. Fine tuning requires some number of H100 cards. Trivial for the brazilian government. Existing brazilian labs are nothing compared to US hyperscalers but they do have enough capacity to fine tune Qwen. Santos Dumont has 248 H100s + 144 Grace Hoppers. That's what makes this hilariously sad. Brazil could have done some good work here, but it just didn't. Brazil merged two models on a workstation.
- motbus3 4mo agoI've worked with infra resources in that region. The prices of hardware never made sense. Not sure if it's taxes or something, but I'm sure a poor city hall won't be able to afford 250 GPUs for R&D in cutting-edge tech.