6 ms·

There's no need to even make such a complicated task. Recently I found that GPT4 can't even reliably create a list of german nouns with a given article (der /

There's no need to even make such a complicated task. Recently I found that GPT4 can't even reliably create a list of german nouns with a given article (der / die / das). It will mess up a simple list - if you ask it to analyse it, it'll be able to tell you that it's wrong. Then you get it to correct the list and it may still be wrong. It can take several iterations to make the list correct. I would have thought this would be a super easy task for it, but apparently not.

wudangmonk 3y ago
I think this task is beyond the capabilities of what GPT4 can handle, this is simply asking too much out of it. For other languages I'm sure it has no problems. https://faculty.georgetown.edu/jod/texts/twain.german.html https://faculty.georgetown.edu/jod/texts/twain.german.html
- deleted 3y ago
  [deleted]
- freehorse 3y ago
  In greek it is also really bad (and makes rather obvious mistakes), in french it seems much better but makes some very obvious mistakes too. To make it interesting, I emphasise that I want nouns that refer to objects only (else it just spits out profession names and stuff like that, which is not interesting). Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.
  - dontupvoteme 3y ago
    >Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge. The strange/sad thing is that despite being "large language models" they're often hypermyopic on English.. I've done some measurements comparing generation between various languages in the prompt and no matter what I do half the time i cannot get them to not include english text or comments in code unless the request is made in japanese, chinese, or a similar very-different-language.
    - Tainnor 3y ago
      It's long been observed (e.g. Emily Bender has written some articles to that effect) that NLP technology underperforms on languages that aren't English, especially when they are significantly different structurally. If you train and evaluate something mostly on a language loke English, you're going to end up with a model that thinks everything works like English, which means among other things very little morphological complexity.
  - avnigo 3y ago
    It's actually quite amusing asking it to give you a list of consonants in which Greek words can end in, and then example words for each. It's pretty much all hallucination and then it tries to gaslight you.
wenc 3y ago
Curious, could you share your prompt? I just tried asking GPT4 (paid) to create a list of German nouns with der/die/das and it managed to do it correctly.
- LouisSayers 3y ago
  "What are the 200 most common nouns in German with the artikel "der"?" I actually didn't realise it was giving me incorrect info until my gf started looking at it! (I was trying to use it to help me learn german)
  - whats_a_quasar 3y ago
    This isn't the same task that you described in the first comment. Did GPT4 include nouns that didn't use the article "der" in its output? Or did it fail to reply with the 200 most common ones?
    - LouisSayers 3y ago
      Apologies, I was trying to give a little context for those that don't know about the three "the's" in German. The list was mostly correct, but yes it added nouns that were not "der" nouns into the list. It then attempted to correct the list and failed at correcting it. In terms of output, it also didn't want to give a list of 200, but I did manage to get a list of around 100 back.
      - dmazzoni 3y ago
        I wonder if it'd help to ask it to write the noun with the article. I just tried it now - I asked it to list the top 100 German "der" nouns, then asked it to repeat the list but with the article. That made it obvious which ones were wrong! It was unwilling to write "Der Jahr" when "Das Jahr" is correct.
  - wenc 3y ago
    I gave GPT4 that prompt and as far as I can tell, the list is correct. GPT answer: It's difficult to determine the "200 most common" nouns for a particular gender in German since the frequency of word use can vary greatly depending on the context (e.g., conversational, academic, technical). However, here's a list of 100 common masculine nouns, which use the definite article "der". der Mann (man), der Tag (day), der Freund (friend), der Vater (father), der Bruder (brother), der Sohn (son), der Monat (month), der Kurs (course), der Tisch (table), der Stuhl (chair), der Garten (garden), der Zug (train), der Platz (place), der Himmel (sky), der Wagen (car), der Brief (letter), der König (king), der Schuh (shoe), der Baum (tree), der Hund (dog), der Bär (bear), der Fisch (fish), der Vogel (bird), der Schüler (pupil), der Regen (rain), der Wind (wind), der Wald (forest), der Berg (mountain), der Fluss (river), der Junge (boy), der Kaffee (coffee), der Lehrer (teacher), der Arzt (doctor), der Student (student), der Polizist (policeman), der Künstler (artist), der Teller (plate), der Löffel (spoon), der Geldbeutel (wallet), der Laptop (laptop), der Schalter (switch), der Fernseher (television), der Schrank (cupboard), der Helm (helmet), der Hut (hat), der Schmuck (jewelry), der Ring (ring), der Sport (sport), der Schaden (damage), der Boden (floor), der Sand (sand), der Sturm (storm), der Preis (price), der Feind (enemy), der Held (hero), der Raum (room), der Morgen (morning), der Abend (evening), der Unfall (accident), der Vortrag (lecture), der Urlaub (vacation), der Ausflug (trip), der Hafen (harbor), der Bericht (report), der Name (name), der Bauer (farmer), der Rechner (calculator), der Traum (dream), der Anzug (suit), der Geschmack (taste), der Eindruck (impression), der Zweck (purpose), der Vertrag (contract), der Krieg (war), der Kunde (customer), der Arbeitgeber (employer), der Mitarbeiter (employee), der Kollege (colleague), der Bewohner (resident), der Fahrer (driver), der Gast (guest), der Kritiker (critic), der Profi (professional), der Sieger (winner), der Kandidat (candidate), der Beamte (official), der Insasse (inmate), der Zeuge (witness), der Beweis (proof), der Schatten (shadow), der Zweifel (doubt), der Trauer (grief), der Frieden (peace), der Nerv (nerve), der Horizont (horizon), der Gedanke (thought), der Lohn (wage), der Antrag (application), der Verlust (loss), der Betrag (amount),
akira2501 3y ago
> I would have thought this would be a super easy task for it Why did you think that? This isn't meant to be critical, but I'm honestly curious, what led you to believe that technology underlying GPT-4 made it a good fit for this or any particular task?
- londons_explore 3y ago
  I too would think it a super easy task. It has probably seen the correct nouns used millions of times in the training data - and asking it to produce the correct nouns for a bunch of words is really just "tell me which case you saw most during training", which is something LLM's are really good at.
  - pmoriarty 3y ago
    From what I understand, it's trained mostly on English, and performs better in English. So it's not at all surprising it would make more mistakes in German or some other language.
- constantcrying 3y ago
  >Why did you think that? It is a purely statistical model. It does not know any "rules" about the language (it doesn't know any language at all), but it is fed data and derives from that sophisticated probabalistical relationships between words. It shouldn't have much of a problem to generate the correct grammatical formulations, as it has been extensively trained on them. Moreso than any other technology neural networks are suited for this kind of tasks where hard rules do not exist (as a German I couldn't tell you why "rain" is masculine, but "machine" is feminine), but lots of data correctly implements the rule, does.
  - Trapais 3y ago
    I have doubts it was extensively trained on German data. Who knows about GPT4, but GPT3 is ~92% of English and ~1.5% of German, which means it saw more "die, motherfucker, die" than "die Mutter". (https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_word_count.csv https://github.com/openai/gpt-3/blob/master/dataset_statisti...)
nielsole 3y ago
With the following prompt i got 50 consecutive words with correct articles: > Erstelle eine Liste mit 10 Nomen, die den Artikel "der" haben. Maybe "reliably" is doing a lot of the heavy lifting?
- LouisSayers 3y ago
  It's not that it didn't do an OK job, but more that you couldn't rely on what it had produced totally, nor rely on it having corrected the list without first having it reanalyse the "corrected" list. It's still extremely helpful, I just found it strange that it seemed like a simple task - for something that has been fed millions of documents, for it to still give some incorrect results - especially AFTER it had analysed its own results and found some noun artikels to be incorrect.
  - afro88 3y ago
    I've found you still can't rely on LLMs to do anything 100% correct without human oversight. Unless you spend a lot of time prompt engineering and testing. Even then you might not get as close to 100% as you'd like. But as you say, they are still extremely helpful anyway.
deleted 3y ago
[deleted]
dontupvoteme 3y ago
Does it work if you ask in German? I found it's a better if you tell it via system prompt that it's a language professor (using your target language) than if you just use english for tasks involving foreign language. The power of the LARP. (I use normal machine translation API for a lot of this, but you can also ask it in another context window to translate the text to other languages. I use this approach for e.g. sindarian)
- LouisSayers 3y ago
  I didn't do that, but will give it another go. Thanks for the suggestion! Otherwise my current strategy is to put it in an analysis loop until it deems the list to be correct.
courseofaction 3y ago
I've gone from working with 500-1k lines of code at a time in GPT-4, to not bothering beyond one or two small functions because it gets confused so easily now. Seems they're enshittifying already, I hope a competitive model is released soon.
- refulgentis 3y ago
  The model didn't change.
  - VeninVidiaVicii 3y ago
    Supposedly. But I have the same experience. I used it to create code to design a complicated CRISPR experiment, but lately it can’t keep anything straight.
    - refulgentis 3y ago
      That should lead you to be less confident that it changed, not more
      - pillefitz 3y ago
        How so? Seems counterintuitive
        refulgentis 3y ago
        I’d expect it based on the absurdity of the difference - it’s a master programmer and now it can’t do anything! - combined with self-knowledge that 0 attempts were made to make an objective comparison. If my Tesla went 80 mph and started going 15, we wouldn’t attribute that to the-nature-of-Teslas, or a software update: there are vast incentives for anyone who knew that was done to share that publicly. Instead, we know something is wrong with the individual car and we take it to the dealer. Here, the missing part is objectivity via a speedometer: I have one, I know outputs are consistent on 0315 models at 0 temperature.
spaceprison 3y ago
I just had the same experience. It was so strange.

wudangmonk 3y ago

I think this task is beyond the capabilities of what GPT4 can handle, this is simply asking too much out of it. For other languages I'm sure it has no problems. https://faculty.georgetown.edu/jod/texts/twain.german.html https://faculty.georgetown.edu/jod/texts/twain.german.html

deleted 3y ago
[deleted]
freehorse 3y ago
In greek it is also really bad (and makes rather obvious mistakes), in french it seems much better but makes some very obvious mistakes too. To make it interesting, I emphasise that I want nouns that refer to objects only (else it just spits out profession names and stuff like that, which is not interesting). Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.
- dontupvoteme 3y ago
  >Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge. The strange/sad thing is that despite being "large language models" they're often hypermyopic on English.. I've done some measurements comparing generation between various languages in the prompt and no matter what I do half the time i cannot get them to not include english text or comments in code unless the request is made in japanese, chinese, or a similar very-different-language.
  - Tainnor 3y ago
    It's long been observed (e.g. Emily Bender has written some articles to that effect) that NLP technology underperforms on languages that aren't English, especially when they are significantly different structurally. If you train and evaluate something mostly on a language loke English, you're going to end up with a model that thinks everything works like English, which means among other things very little morphological complexity.
- avnigo 3y ago
  It's actually quite amusing asking it to give you a list of consonants in which Greek words can end in, and then example words for each. It's pretty much all hallucination and then it tries to gaslight you.

wenc 3y ago

Curious, could you share your prompt? I just tried asking GPT4 (paid) to create a list of German nouns with der/die/das and it managed to do it correctly.

LouisSayers 3y ago
"What are the 200 most common nouns in German with the artikel "der"?" I actually didn't realise it was giving me incorrect info until my gf started looking at it! (I was trying to use it to help me learn german)
- whats_a_quasar 3y ago
  This isn't the same task that you described in the first comment. Did GPT4 include nouns that didn't use the article "der" in its output? Or did it fail to reply with the 200 most common ones?
  - LouisSayers 3y ago
    Apologies, I was trying to give a little context for those that don't know about the three "the's" in German. The list was mostly correct, but yes it added nouns that were not "der" nouns into the list. It then attempted to correct the list and failed at correcting it. In terms of output, it also didn't want to give a list of 200, but I did manage to get a list of around 100 back.
    - dmazzoni 3y ago
      I wonder if it'd help to ask it to write the noun with the article. I just tried it now - I asked it to list the top 100 German "der" nouns, then asked it to repeat the list but with the article. That made it obvious which ones were wrong! It was unwilling to write "Der Jahr" when "Das Jahr" is correct.
- wenc 3y ago
  I gave GPT4 that prompt and as far as I can tell, the list is correct. GPT answer: It's difficult to determine the "200 most common" nouns for a particular gender in German since the frequency of word use can vary greatly depending on the context (e.g., conversational, academic, technical). However, here's a list of 100 common masculine nouns, which use the definite article "der". der Mann (man), der Tag (day), der Freund (friend), der Vater (father), der Bruder (brother), der Sohn (son), der Monat (month), der Kurs (course), der Tisch (table), der Stuhl (chair), der Garten (garden), der Zug (train), der Platz (place), der Himmel (sky), der Wagen (car), der Brief (letter), der König (king), der Schuh (shoe), der Baum (tree), der Hund (dog), der Bär (bear), der Fisch (fish), der Vogel (bird), der Schüler (pupil), der Regen (rain), der Wind (wind), der Wald (forest), der Berg (mountain), der Fluss (river), der Junge (boy), der Kaffee (coffee), der Lehrer (teacher), der Arzt (doctor), der Student (student), der Polizist (policeman), der Künstler (artist), der Teller (plate), der Löffel (spoon), der Geldbeutel (wallet), der Laptop (laptop), der Schalter (switch), der Fernseher (television), der Schrank (cupboard), der Helm (helmet), der Hut (hat), der Schmuck (jewelry), der Ring (ring), der Sport (sport), der Schaden (damage), der Boden (floor), der Sand (sand), der Sturm (storm), der Preis (price), der Feind (enemy), der Held (hero), der Raum (room), der Morgen (morning), der Abend (evening), der Unfall (accident), der Vortrag (lecture), der Urlaub (vacation), der Ausflug (trip), der Hafen (harbor), der Bericht (report), der Name (name), der Bauer (farmer), der Rechner (calculator), der Traum (dream), der Anzug (suit), der Geschmack (taste), der Eindruck (impression), der Zweck (purpose), der Vertrag (contract), der Krieg (war), der Kunde (customer), der Arbeitgeber (employer), der Mitarbeiter (employee), der Kollege (colleague), der Bewohner (resident), der Fahrer (driver), der Gast (guest), der Kritiker (critic), der Profi (professional), der Sieger (winner), der Kandidat (candidate), der Beamte (official), der Insasse (inmate), der Zeuge (witness), der Beweis (proof), der Schatten (shadow), der Zweifel (doubt), der Trauer (grief), der Frieden (peace), der Nerv (nerve), der Horizont (horizon), der Gedanke (thought), der Lohn (wage), der Antrag (application), der Verlust (loss), der Betrag (amount),

akira2501 3y ago

> I would have thought this would be a super easy task for it Why did you think that? This isn't meant to be critical, but I'm honestly curious, what led you to believe that technology underlying GPT-4 made it a good fit for this or any particular task?

londons_explore 3y ago
I too would think it a super easy task. It has probably seen the correct nouns used millions of times in the training data - and asking it to produce the correct nouns for a bunch of words is really just "tell me which case you saw most during training", which is something LLM's are really good at.
- pmoriarty 3y ago
  From what I understand, it's trained mostly on English, and performs better in English. So it's not at all surprising it would make more mistakes in German or some other language.
constantcrying 3y ago
>Why did you think that? It is a purely statistical model. It does not know any "rules" about the language (it doesn't know any language at all), but it is fed data and derives from that sophisticated probabalistical relationships between words. It shouldn't have much of a problem to generate the correct grammatical formulations, as it has been extensively trained on them. Moreso than any other technology neural networks are suited for this kind of tasks where hard rules do not exist (as a German I couldn't tell you why "rain" is masculine, but "machine" is feminine), but lots of data correctly implements the rule, does.
- Trapais 3y ago
  I have doubts it was extensively trained on German data. Who knows about GPT4, but GPT3 is ~92% of English and ~1.5% of German, which means it saw more "die, motherfucker, die" than "die Mutter". (https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_word_count.csv https://github.com/openai/gpt-3/blob/master/dataset_statisti...)

nielsole 3y ago

With the following prompt i got 50 consecutive words with correct articles: > Erstelle eine Liste mit 10 Nomen, die den Artikel "der" haben. Maybe "reliably" is doing a lot of the heavy lifting?

LouisSayers 3y ago
It's not that it didn't do an OK job, but more that you couldn't rely on what it had produced totally, nor rely on it having corrected the list without first having it reanalyse the "corrected" list. It's still extremely helpful, I just found it strange that it seemed like a simple task - for something that has been fed millions of documents, for it to still give some incorrect results - especially AFTER it had analysed its own results and found some noun artikels to be incorrect.
- afro88 3y ago
  I've found you still can't rely on LLMs to do anything 100% correct without human oversight. Unless you spend a lot of time prompt engineering and testing. Even then you might not get as close to 100% as you'd like. But as you say, they are still extremely helpful anyway.

deleted 3y ago

[deleted]

dontupvoteme 3y ago

Does it work if you ask in German? I found it's a better if you tell it via system prompt that it's a language professor (using your target language) than if you just use english for tasks involving foreign language. The power of the LARP. (I use normal machine translation API for a lot of this, but you can also ask it in another context window to translate the text to other languages. I use this approach for e.g. sindarian)

LouisSayers 3y ago
I didn't do that, but will give it another go. Thanks for the suggestion! Otherwise my current strategy is to put it in an analysis loop until it deems the list to be correct.

courseofaction 3y ago

I've gone from working with 500-1k lines of code at a time in GPT-4, to not bothering beyond one or two small functions because it gets confused so easily now. Seems they're enshittifying already, I hope a competitive model is released soon.

refulgentis 3y ago
The model didn't change.
- VeninVidiaVicii 3y ago
  Supposedly. But I have the same experience. I used it to create code to design a complicated CRISPR experiment, but lately it can’t keep anything straight.
  - refulgentis 3y ago
    That should lead you to be less confident that it changed, not more
    - pillefitz 3y ago
      How so? Seems counterintuitive
      - refulgentis 3y ago
        I’d expect it based on the absurdity of the difference - it’s a master programmer and now it can’t do anything! - combined with self-knowledge that 0 attempts were made to make an objective comparison. If my Tesla went 80 mph and started going 15, we wouldn’t attribute that to the-nature-of-Teslas, or a software update: there are vast incentives for anyone who knew that was done to share that publicly. Instead, we know something is wrong with the individual car and we take it to the dealer. Here, the missing part is objectivity via a speedometer: I have one, I know outputs are consistent on 0315 models at 0 temperature.

spaceprison 3y ago

I just had the same experience. It was so strange.