7 ms·
Teuken-7B-Base and Teuken-7B-Instruct: Towards European LLMs (2024)
- smokel 2y agoA paper on languages that begins with a grammatical error in the first sentence does not inspire confidence: > LLMs represents a disruptive technology
- NitpickLawyer 2y agoHey, at least it's not generated by chatgpt :D Funny how LLMs now write cleaner than humans in most cases.
- spacebanana7 2y agoI imagine there was a similar tipping point in the Industrial Revolution where machines started marking "better" manufactured items than artisans.
- InsideOutSanta 2y agoInterestingly, we then collectively decided that, in many cases, imperfect artisanal things were better than perfect industrially produced things. So maybe people will start intentionally putting mistakes into their texts to prove they're not machines. I'm already reluctant to use the em-dash correctly because so many people think only LLMs know how to use it.
- deleted 2y ago[deleted]
- Miraltar 2y agoIt's not that I think only LLMs know em-dashes but they abuse it so much I get annoyed everytime I see one
- spacebanana7 2y agoLLMs seem to use them much more than normal people. If I were a teacher marking homework, em-dashes would be at least an amber flag for LLM use.
- croes 2y agoGiven that it’s about non-English languages it is forgivable
- JKolios 2y agoMore diversity in the LLM space is always good. In my experience though, speaking as a native speaker of one of the less-used European languages, Mistral's models already use it pretty well.
- Etheryte 2y agoAs a native of another small European language, no state of the art model comes anywhere close to not being laughably bad, so more work in this space is definitely welcomed as far as I'm concerned.
- debugnik 2y agoReally? In my experience, Le Chat eventually devolves into spanglish when trying to speak Spanish, so I would have expected worse from Mistral for minority languages.
- isodev 2y agoI live in a country with 3 national languages and I happen to use all of them + English + another one where most of our clients are based. Mistral is the only model atm which doesn’t make a mess of it all. It’s not perfect, but it doesn’t force me to “pretranslate” things.
- kiru_io 2y agoMaybe someone should edit the title to mention this is from 2024: [Submitted on 30 Sep 2024 (v1), last revised 15 Oct 2024 (this version, v2)]
- dang 2y agoAdded. Thanks!
- ozgune 2y agoI had a related, but orthogonal question about multilingual LLMs. When I ask smaller models a question in English, the model does well. When I ask the same model a question in Turkish, the answer is mediocre. When I ask the model to translate my question into English, get the answer, and translate the answer back to Turkish, the model again does well. For example, I tried the above with Llama 3.3 70B, and asked it to plan me a 3-day trip to Istanbul. When I asked Llama to do the translations between English <> Turkish, the answer was notably better. Anyone else observed a similar behavior?
- petesergeant 2y agoFascinating phenomenon. It's like a new Sapir–Whorf hypothesis. Do language models act differently in different languages due to those languages or the training materials?
- shaky-carrousel 2y agoThey absolutely do. They know more in English than in Spanish, I've seen that on all models, since the beginning.
- namaria 2y agoThey have more data in English than Spanish. LLMs don't know or reason or follow instructions. They merely render text continuations that are coherent with the expectations you set when prompting. The fact that they are not able to sustain the illusion in languages with less available training data than English should make that clear.
- shaky-carrousel 2y ago> They have more data in English than Spanish. Yep, that there seems like the definition of knowing. Don't worry, your humanity isn't at risk.
- 2y ago
- miros_love 2y ago>European versions of ARC But this is an image-like benchmark. Has anyone looked at the article about the EU-ARC, what is the difference? Why can't you measure it on a regular one? I glanced through it, didn't find it right away, but judging by their tokenizer, they are learning from scratch. In general, I don't like this approach for the task at hand. For large languages, there are already good models that they don't want to compare with. And for low-resource languages, it is very important to take more languages from this language group, which are not necessarily part of the EU
- whiplash451 2y agoYou might be confusing ARC-AGI and EU-ARC which is a language benchmark [1] [1] https://arxiv.org/pdf/2410.08928 https://arxiv.org/pdf/2410.08928
- Etheryte 2y agoWhy would they want more languages from outside of the EU when they've clearly stated they only target the 24 official languages of the European Union?
- miros_love 2y agoFor example: Slovene language. You simply don't have enough data on it. But if you add all the data that is available on related languages, you will get a higher quality. LLM fails with this property for low-resource languages.
- yorwba 2y agoThey train on 14 billion tokens in Slovene. Are you sure that's not enough?
- miros_love 2y agoUnfortunately, yes. We need more tokens, more variety of topics in texts and more complexity.
- YetAnotherNick 2y agoThey compared with Llama 3.1 and found that to be better on average for their tasks like European MMLU. And Llama 3.1 is the worst in the batch with Qwen 2.5 and Gemma 3 being significantly better.
- KronisLV 2y agoI also quite liked the EuroLLM project: https://huggingface.co/blog/eurollm-team/eurollm-9b https://huggingface.co/blog/eurollm-team/eurollm-9b Was pretty good with Latvian (better than other models this size as well as variants of Llama or Qwen that I could run) and I assume probably with other EU languages as well.
- deleted 2y ago[deleted]
- TheMatten 2y agoI've just tried it in one of the supported languages, and it seems to respond far better than any model under 24B that I've tried before. With its licensing, it sounds much more exciting to me than the OP.
- tannhaeuser 2y agoI mean, Mistral AI is a Paris-based company, and theirs was considered on par or better than other open weight models such as llama3.1 and qwen2.5, and mistral-24b is currently beating oh-so-great gemma3-27b depending on tasks. Also, Stable Diffusion was originally (and still is I believe) developed in Munich. It's true though that raising capital and finding investors works wayyy better in the US (kindof needless to say on HN) and so was getting top talent - at least in the past. Don't get me started on energy prices ;) but I don't believe those contribute significantly in the end anyway.
- deleted 2y ago[deleted]
- nickpsecurity 2y agoYou don't think American companies raising hundreds of millions to ten billion for training models contributed to their model performance or market positions? I think a pile of money and talent is largely the cause of where they're at.
- jug 2y agoOn this topic, don’t miss the quite useful benchmark: https://euroeval.com https://euroeval.com
- anhner 2y agoah, yes... Europe, the continent with 10 countries one of them with 50k population
- jeffhuys 2y agoCould you elaborate on what you wish to convey with this comment?
- supermatt 2y agoI guess its a sarcastic statement about EuroEval covering a fraction of the European languages, yet containing Faroese. It was called ScandEval until recently.
- anhner 2y agoyes, that was my point, thank you. I did not know it was focused on the scandinavian countries until recently, I guess it makes sense now why it has the languages it does.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- NKosmatos 2y agoThere is also a Greek LLM from 2024. Meltemi: A large foundation Language Model for the Greek language https://huggingface.co/ilsp/Meltemi-7B-v1.5 https://huggingface.co/ilsp/Meltemi-7B-v1.5
- pehtis 2y agoMeltemi is ok, but it's "old" and not that good by today's standards. If you need a good Greek local LLM try https://huggingface.co/ilsp/Llama-Krikri-8B-Instruct https://huggingface.co/ilsp/Llama-Krikri-8B-Instruct. Yes, I know it's based on LLama and not a foundation model, but it is still a LOT better than Meltemi.
- htrp 2y agoTIL there are european versions of ARC, HellaSwag, MMLU, and TruthfulQA.
- andai 2y agoCan someone explain this? They just reduce the English text during pretraining to balance it out? Shouldn't that harm every other benchmark though?
- NetOpWibby 2y agoUpset that my mind went, "TEKKEN 7 LLM." Imagine Heihachi Mishima vibe-coding for you.
- denimboy 2y agoI wonder how this compares to RWKV-V5 7B https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers