6 ms·
AMÁLIA and the future of European Portuguese LLMs
- timmers 5mo ago[flagged]
- hartator 5mo agoWhat a waste of time and money. Trying to force a LLM into a specific language makes you missed out on most of the world knowledge.
- mistrial9 5mo ago> makes you missed out on most of the world knowledge and, who knows what will happen to grammar ?
- embedding-shape 5mo agoWhat LLM isn't forced into a specific language? That'd be a weird language model no one could understand, you need to chose at least one language, ideally the same as the creators speak. Besides, there is knowledge that is locked behind languages, there are things known in Portuguese that aren't known in other languages, and the same for other languages too. More accessibility to those ideas wouldn't hurt.
- Miraste 5mo agoTo my knowledge, all major LLMs are multilingual. This article could really have used an evaluation of existing models' European Portuguese capabilities.
- deleted 5mo ago[deleted]
- cess11 5mo agoE.g. gemma3:4b can fake simple conversations in several european languages, including portuguese, swedish and finnish. It's just a database. If you push text in one language into it, it'll likely crap out stuff in that same language, unless the system prompt that also goes in with your query causes it not to.
- numpad0 5mo agoyeah, they seem all confined to being an American-consultant-Chinese-authoritarian split personality with broad second language capabilities. I suppose they become too incoherent otherwise.
- clear-octopus 5mo ago[dead]
- KK7NIL 5mo ago[flagged]
- lmf4lol 5mo agoeveryone on this project probably learned a lot doing it, dont you think!
- joe_mamba 5mo agoI'd also want to get paid to work on stuff not meant to bring any financial returns to my employer, just to learn and pad my resume. Sounds like a sweet gig. Where do I sign up?
- lmf4lol 5mo agobecome a scientist? apply for research grants. Join an NGO. Lots of options Not everything has to return money.
- joe_mamba 5mo ago>become a scientist? apply for research grants. Join an NGO. You mentioned only 2(grants are also academia) and most NGOs don't hire SW devs. That's hardly "lots", basically none out of academia. >Not everything has to return money. But then where does that money they give you come from?
- xp84 5mo agoIt's the European Way
- schnitzelstoat 5mo agoIt reminds me of Qwant - the French Google alternative that got a load of public money. The fact you've probably never heard of it shows how well that went.
- CrimsonRain 5mo agoEurope always has a thing for their languages. They think many languages make them stronger while spending billions in system loss due to communication barriers. It is obvious they will try to do the same with LLMs and call it the next best thing since bread and butter. I went to JCON EUROPE this year. The amount of "Europe this" "Europe that" "sovereign this, sovereign that" is mind boggling and just a waste of time and money. The regular people know this and thus left the conferences mid way. But somehow the people "in charge" really need to push this. Same thing here.
- lmf4lol 5mo agowhats your suggestion? we just eradicate all of our culture and languages and go full on english ? whats wrong with exploring ways to keep national languages alive in the LLM area
- joe_mamba 5mo ago>and go full on english ? Nobody is saying you have to swap your culture for English. You can have English as the mandatory language for tech and business across the EU, while still keeping your language and culture for your education, leisure, festivities, art, media, etc. This way everyone is happy. But countries like France would rather detonate its entire nuclear arsenal rather than accepting official use of English on its own soil. As long as resources are spent across the EU to account for every language and bureaucracy, we'll keep falling behind internationally, and the only winners will be the bureaucrats, notaries, lawyers, consultants, translators, etc. which would be fine if this were preserving culture like you said in the beginning, but it isn't, it's just preserving friction, segmentation and bureaucracy. We need another Concord moment. What's wild is that Concord was made via international cooperation, before the EU was even a thing. So whatever the EU is doing to improve things, it's either not good, not enough, or not working. I hope this improves but knowing how petty some EU states are about things being done their way, I doubt it.
- schnitzelstoat 5mo agoThere's an obvious advantage to everyone speaking the same language - although perhaps real-time translation with LLMs and hardware like the Timekettle will reduce this problem. Personally, I wouldn't really care if that language were English or Mandarin Chinese tbh. Training an entire LLM model for each language is going to be incredibly expensive and likely a waste of resources. Keep in mind that all the big LLMs can already speak these languages anyway - this effort is just to make a 'pure' Portuguese LLM.
- swiftcoder 5mo agoIt is definitely an interesting problem, because Portugal is a small enough country that the actual total corpus of available texts in (non-Brazilian) Portuguese is potentially problematic.
- embedding-shape 5mo agoI don't think so, Portugal the country might be small, with a small population, but there is ~250 million "Lusophones" (native Portuguese speakers), making it the fifth-most spoken native language in the world, I'd hardly call that small :) And before everyone screams; yes, European Portuguese is different from Brazilian Portuguese, but they're still both Portuguese and understand each other, so it's not like the text from one cannot be used to train a model for the other, or vice-versa. All in all, I don't think that's a major issue here.
- KK7NIL 5mo agoThe whole point of this project is to have an LLM that speaks European Portuguese, not Brazilian Portuguese.
- embedding-shape 5mo agoRight, and my point is that if you use 80% Brazilian Portuguese during base model training + 20% European Portuguese as post-training, you pretty much get exactly that, except with a ton more of available training data.
- KK7NIL 5mo agoWhat's your evidence for that? And if the first 80% doesn't bias the language after post-training (which I think is what you're claiming) why not go for English or a mixture of languages, which is essentially what they did by starting with EuroLLM?
- embedding-shape 5mo ago
- pu_pe 5mo agoI'm not sure the direction should be to finetune a small local model for each country or language. These models are already not particularly great at information retrieval, so I doubt anyone would use them for questions like the author suggests (ie who was the president between X and Y). Similarly, they are a little too lightweight to be used for translations too. If the budget is indeed so modest (5.5 million euros!), I would focus completely on preparing datasets and making sure all open cultural artifacts that we can find are well documented in them. That way every model, private or open, that gets trained in the future could better represent the culture and language of your country.
- dyauspitr 5mo agoYeah I think India is going the better route with Sarvam which is trained from scratch and still relatively cheap.
- iugtmkbdfil834 5mo agoI agree, the research is complex enough as is without having to worry about splitting it babel-like into multiple languages.
- TheMagicHorsey 5mo agoThis is the way. Sovereign SOTA models might also be possible with nation-state involvement. But this is a good stopgap.
- dudefeliciano 5mo ago> who was the president between X and Y this is the type of question that should never ever be asked to an llm running on some A100 on the other side of the world, local llms are already more than capable to answer these
- algoth1 5mo agoWouldnt it be easier to fine tune a model to convert the Brazilian Portuguese corpus into European Portuguese and then use that corpus?
- kinow 5mo agoThat idea is different than what most are talking here in other comments. The grammar and vocabularies don't match, but I think the worst are the expressions. Both sides have *a lot* of expressions that vary per context and location.
- algoth1 5mo agoI’m a native Portuguese speaker, whom grow up watching morning Brazilian Portuguese cartoons. I would say it’s doable
- simianwords 5mo agoDomain specific models will never be a thing. You don't get generalised intelligence with that. https://simianwords.bearblog.dev/why-domain-specific-llms-wont-exist-an-intuition/ https://simianwords.bearblog.dev/why-domain-specific-llms-wo...
- mariopt 5mo agoThis model is a waste of Public Funds. There is no public website to use it, be it free or paid, the dataset is not public, the code is not public (The github URL in the article returns 404 ), the claimed model intelligence is so low that is pretty much useless at 32K context and massively inferior to GPT‑4o. As per tradition in Portugal, some people managed to get 5.5 Million to produce nothing and no one is asking questions. You want a better idea? Just fine tune the open source Kimi 2.6 with an open source Portuguese dataset, the cost would be under a million and we would be getting something useful. It would be really nice to know what happened to 5.5 Millions whilst not being able to even provide a functional website to use the model.
- dr_dshiv 5mo agoIt’s a way to suck all the money out of the room in the name of nationalism — and it’s all over Europe. Only idea everyone has had.
- vova_hn2 5mo agoI'm not arguing with the rest of your points, but... > Just fine tune the open source Kimi 2.6 with an open source Portuguese dataset I think that tokenizers of all popular models are heavily biased towards English or English and Mandarin. And I don't think that it is possibple to replace the tokenizer without full retraining.
- mcyc 5mo agoYou are right about most tokenizers being heavily biased towards English, but the situation is not so bad for Portuguese. Here are some results on the Goldfish corpus [1] with a few different tokenizers. This measures #characters in corpus / #subwords in tokenized corpus. ``` Llama3 english, 0.216 portuguese, 0.285 italian, 0.287 greek, 0.592 ``` ``` Gemma4 english, 0.219 portuguese, 0.246 italian, 0.249 greek, 0.537 ``` ``` Kimi2.6 english, 0.214 portuguese, 0.310 italian, 0.308 greek, 0.716 ``` Portuguese is worse than English certainly, but it is on par with Italian (which I think has more overlap with English) and much better than Greek (since it doesn't use the Latin script and is definitely not prioritized in the tokenizer construction). On your second point, tokenizer transfer allows for extending/modifying a tokenizer without retraining the model from scratch. The simplest version of this is tokenizer extension + continual pretraining, where you just add a bunch more tokens to the vocab for the language/domain that you want to improve and train a little more. It's been done for Japanese [2] and Indic languages, but afaik not Portuguese. So I think that continual pretraining for a large base model would have probably been fine for this case with huge cost savings. But it is good to have the ability to train your own base models, so I don't think this is such a bad idea. ----------------------- [1]: https://huggingface.co/datasets/goldfish-models/fish-food https://huggingface.co/datasets/goldfish-models/fish-food [2]: https://arxiv.org/abs/2404.17790 https://arxiv.org/abs/2404.17790
- mt_ 5mo ago5 million for a llama-2 finetune, how is that impressive?
- r2ob 5mo ago"This model is a waste of Public Funds". There is no "public funds", this is a waste of money from the tax payers.
- alexaholic 5mo agoThe Amália model is not yet publicly available. Until it's ready, one can fool around with Anália at https://analia.pt https://analia.pt
- drivebyhooting 5mo agoI’ve noticed that ChatGPT is noticeably dumber in languages other than English. It even will confidently repeat common but wrong superstitions from the target language as if they were fact.
- guilhas 5mo agoAmazing. Every country should have a foothold on AI, as it will have impact every area of a citizen's life Quite cheap compared with most public spendings Europe countries already produce very little. Let's not let the wave pass and end up in a future where Europe is continuously reliant on US and Chinese tech as usual. And their definitions of what truth is