6 ms·
Viking 7B: open LLM for the Nordic languages trained on AMD GPUs
- dmichulke 2y agoIs there something similar for romance or Germanic languages? And how did they decide that, e.g., German or Dutch would make the model worse?
- frodo8sam 2y agoI don't think they decided that, they included Finish which is completely unrelated to the other nordic languages. If they just picked languages that are related for cross learning including Dutch or German would have made more sense indeed.
- coffeebeqn 2y agoThe root is unrelated but Finnish has certainly been shaped by Swedish and Russian and most recently English languages in the last 200 years
- Jensson 2y agoIncluding Finnish was probably just a political choice, since Finland and Sweden are very close politically, much closer than Germany or other areas with more similar languages.
- Asraelite 2y agoThis was done by a Finnish company and university. They would've included Finnish even without any political motivation.
- chymist 2y agoThe company is based in Finland, they started with Finnish
- frodo8sam 2y agoI understand, not saying they did something wrong just pointing out the selection of languages was not because of them belonging to the same family but rather to serve a certain region.
- KeplerBoy 2y agoThe nordic languages are germanic except for finnish, but yeah finnish is an exception and I'd expect most small LLMs to struggle with it.
- bangaladore 2y agoI have had this question. How much better would common LLMs (Llama, GPTN) be if they were only trained in one language? I have to assume they would perform better, but I might be wrong.
- fermuch 2y agoJust like adding code to textual models helps the model develop its reasoning capabilities, it seems like adding more languages helps in other areas too. What is needed is more good quality data to train on...
- nickpsecurity 2y agoWe also see humans get worse at specific things when they learn too much in general. There is a cut-off point to how many concepts we can learn with what skill. To be most effective, we have to specialize in the right things while continuing to acquire generalist knowledge. It’s a balancing act. These architectures are less capable than brains in many ways. So, we should expect them to have such trade-offs. An efficient one should work fine on English, mathematical notation, and a programming language. Maybe samples of others that illustrate unique concepts. I’m also curious how many languages or concepts you can add to a given architecture before its effectiveness starts dropping.
- worldsayshi 2y agoI guess you mean non-textual data then because the amount of text data they are being trained on ought to be enough for agi by now? Some kind of diminishing returns asymptote from text volume alone must have been hit a long time ago.
- imtringued 2y agoIt's not the amount that is wrong, it's how the model is trained. The model is trained for zero and few shot tasks. It is not surprising that it is performing well when you ask for that.
- darby_eight 2y ago
- jarbus 2y agoWould love to know more about their experience training on AMD GPUs. Was it just as seamless as using Cuda?
- deleted 2y ago[deleted]
- ganzuul 2y ago> To leverage the capabilities of MI250X, ROCm enables the use of GPU matrix cores through its rocBLAS and MIOpen library implementations that, in turn,are leveraged by PyTorch. - https://aclanthology.org/2023.emnlp-main.164.pdf https://aclanthology.org/2023.emnlp-main.164.pdf https://github.com/TurkuNLP/ https://github.com/TurkuNLP/
- imtringued 2y agoThey probably got a lot of hand holding from AMD.
- KeplerBoy 2y agoHaving access to enterprise GPUs on one of the biggest HPCs systems in Europe is probably enough. AMDs bad rep in AI is mostly due to flaky support of its consumer GPUs.
- ganzuul 2y agoGreat talking points. These are highly relevant subjects and I'm delighted we in the Nordics are keeping up with current developments. This work is important for preserving our culture. I hope to see this used to generate a customized curriculum for each neurodiverse child so that we can live in a more equitable society.
- ChrisArchitect 2y agodouble slash in the shared link probably not ideal (though inconsequential) https://www.silo.ai/blog/viking-7b-the-first-open-llm-for-the-nordic-languages https://www.silo.ai/blog/viking-7b-the-first-open-llm-for-th...
- melenaboija 2y agoAlthough not nordic not including basque which I guess could also be considered an European low-resource language.
- ghnws 2y agoI got the impression they are focusing on the nordic culture as much as the languages. >Silo AI and TurkuNLP are dedicated to developing models that not only excel in linguistic performance and inclusivity but are also attuned to local values and cultures.
- matsemann 2y agoWould an LLM trained on a smaller language have better cultural awareness etc than one trained in English? Because English is written all over the world by all kinds of people, an English LLM will average that (and for instance feel a bit off for an American). But a Norwegian LLM for instance, trained on a language mostly written by Norwegians, would that feel more natural to me in comparison?
- deleted 2y ago[deleted]
- smokracek 2y agoFirst thing I notice is that Finnish is part of a completely different language family from the other Nordic languages and English (Uralic vs. Indo-European). I wonder to what extent this affects the effectiveness of their low-resource training. Finnish is highly agglutinative, adding prefixes and suffixes to modify a root. My (amateur) take is that the tokenization and attention patterns may differ a lot? Would love to see more educated people than I discuss this.
- ghnws 2y agoThen again the culture of Finland is very similar to the other nordics, which looks to be one of the reasons for the project.
- sandworm101 2y ago>> to what extent this affects the effectiveness of The correct use of those words demonstrates that you are either not an AI, all of them being trained on so much bad language, or are an AI from a more perfect future.
- anewhnaccount3 2y agoFinnish is not so different dispute having different lineages. Even if we talk about morphology, sometimes it's simply that e.g. prepositions are affixed to the end of a word big whoop. There are many dimensions to language vairation. Finnish has a long history of contact with Scandi languages and a lot of borrowed words and logic. It would be good to have Estonian and possibly Baltic languages too. ETA: It is differentof course just perhaps not as much as people sometimes try to say. You can definitely ruffle some feathers with this one given the uniqueness of Finnish is pretty central to Finnish nationalism.
- Tor3 2y agoAs someone growing up in relatively close contact with Finnish, I can assure you that there's no real common ground between Scandinavian languages (Swedish, Danish, Norwegian) and Finnish. There are loan words, but they are few and far between and in any case does not make for any mutual understanding. I've been so much to Finland that I would really like to learn to at least understand the language, instead of relying of memorized names of foodstuff and the like. Just have to tackle Japanese first.. (and I consider that one an easier operation)
- halgir 2y ago> extends to include Danish, Finnish, Norwegian, Icelandic, Swedish * cries in Faroese *
- jug 2y agoIf you're interested in this, don't miss AI Sweden's GPT-SW3 @ 126M to 40B trained on Nordic languages (not Finnish) and English. It's funded by the Swedish government and partners, and freely available with a pretty lively Discord for ongoing AI research focusing on the Nordic languages. I think Viking is called "first" because it includes Finnish, because otherwise, GPT-SW3 was released earlier. https://huggingface.co/AI-Sweden-Models https://huggingface.co/AI-Sweden-Models
- lostmsu 2y agoWhy do they do training from scratch instead of starting off LLAMA 3 or something else?
- Bedon292 2y agoI cannot seem to find a link to the actual model from this page or anywhere on the website. This appears to be it: https://huggingface.co/LumiOpen/Viking-7B https://huggingface.co/LumiOpen/Viking-7B
- larodi 2y agoThe fact it was trained on HPC which covers 20% heat consumption in a city is absolutely wild and on par with how wild it is to have English/Nordic model. “ Further emphasizing digital sovereignty, Viking is trained on the EuroHPC supercomputer LUMI, utilizing up to 4096 AMD MI-250X GPUs. LUMI is not only Europe’s most powerful supercomputer and the 5th most powerful in the world, but also the 3rd greenest supercomputer among the top 500 supercomputers. LUMI’s energy consumption is covered with power produced 100% with hydroelectricity, and the waste heat of LUMI will account for about 20 percent of the district heating in the surrounding city of Kajaani. ”