8 ms·
LeoLM: German-Language LLM Research
- petesergeant 3y agoI'd be so interested in seeing how different the world views are of a model fine-tuned on English Wikipedia vs one fine-tuned on translated German Wikipedia.
- intellectronica 3y agoI don't get it - is standard Llama bad at German? Other LLMs which I use regularly (like GPT-3.5 and GPT-4) work just find in German out of the box, as well as in several other languages.
- antupis 3y agoAt least with Finnish GPT-3.5 and GPT-4 are only models that really work other are rather horrible.
- anonylizard 3y agoOf course. The larger the model is (GPT3.5+), the more languages it can hold in memory, so they are naturally multilingual. But small models (Llama) need dedicated training to be good at another language. Its similar to how high verbal IQ people can easily master multiple languages, but normal people are only good at their main language.
- KeplerBoy 3y agoThat last paragraph is a bold claim, my friend.
- benreesman 3y agoI’ve heard (credible-sounding) gossip that most folks hit a 1.5Tw 16-way mix on GPT-4. Is that like, consensus these days? A couple of months ago I got 3.5-turbo to start talking about “the details” but that is about as reliable as anything it says. It was claiming like, 200Gw. Are solid numbers available?
- Hendrikto 3y ago> A couple of months ago I got 3.5-turbo to start talking about “the details” but that is about as reliable as anything it says. Less reliable, probably. Very unlikely this information is in the training set.
- softg 3y ago> Its similar to how high verbal IQ people can easily master multiple languages, but normal people are only good at their main language. How? Are you implying "high verbal IQ people" have bigger "models" in their heads somehow?
- ocimbote 3y ago
- famouswaffles 3y agoThose models (3.5 and 4) are presumably trained on enough german text to be fluent. Llama is not.
- speedgoose 3y agoStandard Llama is bad at everything but English. It’s mostly trained on English and targeting English.
- messutied 3y agoMaybe a dumb question but, are there advantages of using Llama instead of ChatGPT besides cost and the fact that it’s open?
- speedgoose 3y agoWhen you publish a model, it’s better to publish the weights. You cannot access the weights of ChatGPT fine tuned models, which is a dealbreaker.
- codeflo 3y agoPerhaps “As a large language model, I’m unable to…”?
- jug 3y agoI’ve been thinking along these lines and if it’ll become more common with smaller but more tailored language models? The advantage seems obvious in that they would have significantly lower resource requirements both as for prediction and fine tuning, which matters when companies dealing with sensitive data (or even just comply with strict EU laws) wants to run them locally. It makes sense that this specialization is seen first with language coverage because only GPT-4 is often considered truly fit for professional multilingual use; an LLM that is expensive and in the cloud.
- kohlerm 3y ago[flagged]
- MenhirMike 3y ago[flagged]
- usr1106 3y agoSo this seems completely open source. Naive question: If I wanted to run this at home just for fun (basically one slightly patient user) what kind of machine(s) I would need?
- redox99 3y agoYou can run a 7B and 13B quantized model on almost any machine on the CPU if you're patient.
- dang 3y ago[stub for offtopicness]
- awestroke 3y ago[flagged]
- foota 3y ago[flagged]
- Helmut10001 3y ago[flagged]
- endymi0n 3y agoHow many German LLMs do you need to change a lightbulb? One. They're efficient and don't have humor.
- bhawks 3y agoIt is really just a language model but because German sentences are about 4 times larger it is technically a large language model.
- akumetsu 3y agoCool to see this but I am always surprised how often LLM output is used to train other LLMs. They used gpt3.5 turbo and gpt4 for multiple tasks. Even simple translation of englisch benchmarks and writing German poems to train on, in order to create an LLM that works better in German? AFAIK, this still goes against OpenAI TOS and also the basic idea that training on AI output leads to worse results in general. Was there some major shift in this over the years, or has it simply become the default approach due to it being easy to do? Also nice to finally see something from Hessian.AI, as a local, I heard them talk big more than once but never saw results. I wonder what Aleph Alpha thinks about this, since they want to make "AI made in Europe to challenge OpenAI"
- tosh 3y agoOut of the box it seems to generate very long "Schachtelsätze". I wonder how english models manage to avoid this (system prompt? different training material mix?). That said: similar to other large language models you can get it to generate different styles of text. For example "antworte auf Wienerisch" did change the writing style.
- usr1106 3y agoMe: > Was für Leute trifft man auf Hacker News? Sind Hacker nicht gefährlich? LeoLM 13b chat: > Hacker News ist eine Website, auf der Menschen über Technologie, Startups und alles diskutieren, was mit dem Thema Informatik zusammenhängt. Es ist nicht ungewöhnlich, dass Menschen mit verschiedenen Hintergründen und Interessen Hacker News lesen und beitragen. Daher trifft man auf Hacker News Menschen mit einer Vielzahl von Berufsbezeichnungen, Erfahrungsstufen und Interessen. Hacker allein sind keine gefährlichen Personen, sondern Menschen, die eine Faszination für die Arbeit mit Computern haben. Hacking ist einfach eine Möglichkeit, Computerprogramme zu testen und zu manipulieren, um herauszufinden, wie sie funktionieren. Es gibt jedoch einige wenige Menschen, die diese Fähigkeiten nutzen, um anderen zu schaden. Diese Individuen sind jedoch nicht die Mehrheit.
- derrasterpunkt 3y agoThat is a quality answer.
- usr1106 3y agoIt is. But how do you know that if your are not familiar with the topic already? It will give complete nonsense answers that look equally elegant. Naive question (I don't work with LLMs). Is there any research/figures/measures about how many of the answers can be considered factually correct?
- Tainnor 3y agoThe tone of that answer seems totally off to me. It reads like a school essay of the less inspired variety.
- martin_a 3y agoAlso I find the description of what hacking is rather limited. At least in my understading hacking also applies to hardware and "stuff" in general, not only to "computer programs".
- lagrange77 3y ago
- kebsup 3y agoThis looks great! I'm currently building something like a "gpt powered Anki" in which you can infinitely generate sentences for each word you are learning, but German gpt sentences are sometimes grammatically incorrect. Not the best for language learning app.
- adt 3y agoSeptember 2023 was a massive month for LLM releases. I see 14 big highlights including the LeoLM family, and we haven't even finished the month yet... Models table: https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/ Models timeline: https://lifearchitect.ai/timeline/ https://lifearchitect.ai/timeline/
- usr1106 3y agoThe German seems nearly free of mistakes. The factual content can with a high probability be utter nonsense, no difference from ChatGPT. > Steht das Verb in deutschen Sätzen immer am Ende wie Mark Twain behauptet? > Jennifer Lopez' zweites Studioalbum wurde 1998 veröffentlicht und heißt "On the 6". > Diese Antwort war wohl "Thema verfehlt"! > Das stimmt, ich habe Ihre Frage nicht vollständig beantwortet. Ich entschuldige mich dafür. Deutsche Sätze müssen nicht immer mit dem Verb am Ende geschrieben werden, aber es ist üblich. Die Reihenfolge Subjekt-Verb-Objekt (SVO) ist die häufigste Wortreihenfolge im Deutschen, obwohl sie nicht immer eingehalten wird. Im Deutschen gibt es mehrere mögliche Satzstrukturen und Subjekt-Verb-Objekt ist nur eine davon. Zum Beispiel können Sätze mit Infinitivkonstruktionen oder Partizipialkonstruktionen auch allein mit einem Nomen und einer Präposition beginnen. Zusätzlich können die Reihenfolge der Wörter innerhalb von Sätzen variieren, solange die grundlegende Satzstruktur erhalten bleibt. > Beschränken wir uns mal auf Aussagesätze und Hauptsätze. Wie sind die Regeln für die Stellung des Verbs in diesem Fall? > Im Deutschen steht das Verb im Aussagesatz normalerweise am Satzende (Endstellung), außer bei Fragen und Befehlen, wo das finite Verb (Verbform, die Person und Numerus anzeigt) am Anfang steht. Im Aussagesatz wird die Verbposition am Satzende als Standardverbposition bezeichnet. Diese Standardverbposition gilt jedoch nicht immer und kann auf verschiedene Weise abweichen. Zum Beispiel: 1. In Fragesätzen und Befehlsätzen steht das finite Verb normalerweise am Anfang, vor dem Subjekt. 2. Im Passiv steht das Verb normalerweise am Satzende, aber die Form des Verbs ändert sich je nach Zeitform, Person und Numerus. 3. Die Kopulaverben 'ein', 'bleiben', 'werden', 'heißen' und'sehen' stehen manchmal am Anfang des Satzes, insbesondere in einfachen Sätzen oder Satzteilen. 4. Einige Verben wie 'üssen', 'ollen', 'wollen', 'können' und 'ögen' können am Anfang des Satzes verwendet werden, insbesondere in Befehlsätzen. 5. Reflexivpronomen und Partizipien können mit dem Verb auftreten, das normalerweise am Satzende steht, und sich auf dasselbe Nomen im Satz beziehen. 6. Im Perfekt wird das Hilfsverb 'haben' /'sein' normalerweise vor dem Partizip Perfekt platziert, es sei denn, der Satz enthält ein Hilfsverb. Using LLMs for anything else than writing poems is irresponsible!
- johnnyworker 3y agoWell, it just goes to show that if you already know the answer, you can use LLM, but then it's also pointless. If you don't know the answer, and use an LLM, you may think you learned something but instead now know less.
- usr1106 3y agoResponse times were great 20 minutes ago. Hugged to death right now.
- Beijinger 3y agoWould be interesting to know what is more difficult for AI to learn. German has a more flexible word order, this makes it more difficult for some speakers. "Man bites dog" and "dog bits man" have two different meanings based on word order. In German, both orders are possible and it still has the same meaning.
- MaKey 3y ago> "Man bites dog" and "dog bits man" have two different meanings based on word order. In German, both orders are possible and it still has the same meaning. That's not correct. Both orders are possible but they don't have the same meaning. The word order in this example has the same effect in German as it has in English.
- Beijinger 3y agoDer Mann beisst den Hund. Den Hund beisst der Mann. Both sentences have the same meaning. Beisst der Mann den Hund? would make it a question. "Der Mann den Hund beisst" is grammatically wrong but still understandable. I googled it and, strange coincidence, they even use my example: https://www.dartmouth.edu/~deutsch/Grammatik/WordOrder/WordOrder.html https://www.dartmouth.edu/~deutsch/Grammatik/WordOrder/WordO... English tends to rely mostly on word order to indicate the grammatical function of a word or phrase, while German uses inflections. The German endings, such as those indicating the nominative, accusative, dative, and genitive cases in three different genders, allow for some greater flexibility in clause construction. Hence "Der Hund beißt den Mann" and "Den Mann beißt der Hund" both mean "The dog bites the man" (as opposed to "The man bites the dog"). Nevertheless, German word order is extremely important, even when it is not vital to meaning, and its correctness plays a major role in how a foreigner's command of the language is evaluated. At the same time, word order is an infinitely complex aspect of language, never wholly mastered by non-native speakers. Very few rules cover all possibilities, and context often trumps other considerations. When Robert Frost writes, "Something there is that doesn't love a wall," it's poetic; if someone with a foreign accent says the same thing in conversation, it sounds like Yoda.
- 3y ago
- rob74 3y agoAfter reading the name, I thought that it was related to https://www.leo.org/ https://www.leo.org/ - which initially started out (decades ago) as a project of TUM (Technical University of Munich), but I guess the lion reference is because Hesse also has a lion in its crest, same as Bavaria...
- penguin_booze 3y agoThere's this German-English dictionary https://dict.leo.org/ https://dict.leo.org/. I'm wondering what's the Leo connection with German.
- warpspin 3y agoNone, pure coincidence. leo.org actually comes from "link everything online" and the use of the lion in the Bavarian coat of arms. It used to be a FTP site which transformed to an HTTP archive site, which transformed, if my memory does not betray me to some kind of web directory. Then a couple of students added diverse helper scripts, among those a dictionary and nowadays only the dictionary service survived and was commercialized.