5 ms·

Obvious thought that I haven't tested: can you literally achieve this by getting it to answer in Dutch, or training an AI on Dutch text? Plausibly* Dutch-langua

Obvious thought that I haven't tested: can you literally achieve this by getting it to answer in Dutch, or training an AI on Dutch text? Plausibly* Dutch-language training data will reflect this cultural difference by virtue of being written primarily by Dutch people. * (though not necessarily, since the Internet is its own country with its own culture, and much training data comes from the Internet)

zoover2020 2y ago
I've tried Dutch answers and it is more than happy to hallucinate and give me answers that are very "American". Doesn't help that our culture is very inspired by the US pop culture as well since the internet. Haven't tried prompt engineering with the Dutch stereotype, though.
berkes 2y ago
That hardly works. Though from my limited experiments, claude's models are better at this than OpenAIs. OpenAI will, quite often, come with suggestions that are literal translations of "anglicist" phrases. Such as "Ik hoop dat deze email u gezond vindt" (I hope this email finds you well), which is so wrong that not even "simple" translation tools would suggest this. Seeing that OpenAIs models can (could? This is from a large test we did months ago) not even use proper localized phrases but uses American ones, I highly doubt it can or will respond by refusing answers when it has none based on the training data.
- eru 2y ago
  See https://chatgpt.com/share/67cfaa34-78fc-8009-816f-ca5778c93ef6 https://chatgpt.com/share/67cfaa34-78fc-8009-816f-ca5778c93e... and https://claude.ai/share/d03ce2bf-0bbd-4b91-9afd-eabfc713e99a https://claude.ai/share/d03ce2bf-0bbd-4b91-9afd-eabfc713e99a With ChatGPT O1: https://chatgpt.com/share/67cfaa7e-70ac-8009-871b-571924b5a569 https://chatgpt.com/share/67cfaa7e-70ac-8009-871b-571924b5a5... and with Claude's 'extended': https://claude.ai/share/4ba55410-98f3-4b53-9540-219acd2cdc4c https://claude.ai/share/4ba55410-98f3-4b53-9540-219acd2cdc4c
  - berkes 2y ago
    Some of the examples are still wrong. Nuanced, but a Dutch native will still frown at it. But more importantly is that you limited the context a lot. As in: the scope, the prompt, is very narrow. In our case, we were generating emails. Lines like greetings are but one of 20+ details in that mail and not even the most important ones. The prompts ever larger, the multishot examples ever more tuned. And then, one in a few hundred will turn up with these "horrible" translations. We've now moved to a chain of models, where we generate emails in American (the creative part) and then use another model to translate them to Dutch (the non-creative but culturally aware part). This works much better as we can pick models that are good at one thing or tuned to do this one thing better (either by the LLMAAS provider, or by parameters such as temperature).
    - eru 2y ago
      > Some of the examples are still wrong. Nuanced, but a Dutch native will still frown at it. Thanks, that why I posted the links here: I couldn't judge by myself how good these creations were. > But more importantly is that you limited the context a lot. As in: the scope, the prompt, is very narrow. Yes, I just did a very crude experiment.
pishpash 2y ago
Internal vectors aren't in any particular language.
- daemonologist 2y ago
  No, but theoretically, blunt responses might be more common in Dutch-language training data. A well-fit model would be expected to replicate that. (Basically similar to straight up asking it to be more blunt, except it has probably trained a lot more on _Dutch_ than on _someone just told me to be blunt_ so the effect might be more natural and more subtle.)
daemonologist 2y ago
I wonder (if this works at all) if the effect might be stronger if you also prompted in Dutch, preferably written by a fluent speaker rather than machine-translated.

zoover2020 2y ago

I've tried Dutch answers and it is more than happy to hallucinate and give me answers that are very "American". Doesn't help that our culture is very inspired by the US pop culture as well since the internet. Haven't tried prompt engineering with the Dutch stereotype, though.

berkes 2y ago

That hardly works. Though from my limited experiments, claude's models are better at this than OpenAIs. OpenAI will, quite often, come with suggestions that are literal translations of "anglicist" phrases. Such as "Ik hoop dat deze email u gezond vindt" (I hope this email finds you well), which is so wrong that not even "simple" translation tools would suggest this. Seeing that OpenAIs models can (could? This is from a large test we did months ago) not even use proper localized phrases but uses American ones, I highly doubt it can or will respond by refusing answers when it has none based on the training data.

eru 2y ago
See https://chatgpt.com/share/67cfaa34-78fc-8009-816f-ca5778c93ef6 https://chatgpt.com/share/67cfaa34-78fc-8009-816f-ca5778c93e... and https://claude.ai/share/d03ce2bf-0bbd-4b91-9afd-eabfc713e99a https://claude.ai/share/d03ce2bf-0bbd-4b91-9afd-eabfc713e99a With ChatGPT O1: https://chatgpt.com/share/67cfaa7e-70ac-8009-871b-571924b5a569 https://chatgpt.com/share/67cfaa7e-70ac-8009-871b-571924b5a5... and with Claude's 'extended': https://claude.ai/share/4ba55410-98f3-4b53-9540-219acd2cdc4c https://claude.ai/share/4ba55410-98f3-4b53-9540-219acd2cdc4c
- berkes 2y ago
  Some of the examples are still wrong. Nuanced, but a Dutch native will still frown at it. But more importantly is that you limited the context a lot. As in: the scope, the prompt, is very narrow. In our case, we were generating emails. Lines like greetings are but one of 20+ details in that mail and not even the most important ones. The prompts ever larger, the multishot examples ever more tuned. And then, one in a few hundred will turn up with these "horrible" translations. We've now moved to a chain of models, where we generate emails in American (the creative part) and then use another model to translate them to Dutch (the non-creative but culturally aware part). This works much better as we can pick models that are good at one thing or tuned to do this one thing better (either by the LLMAAS provider, or by parameters such as temperature).
  - eru 2y ago
    > Some of the examples are still wrong. Nuanced, but a Dutch native will still frown at it. Thanks, that why I posted the links here: I couldn't judge by myself how good these creations were. > But more importantly is that you limited the context a lot. As in: the scope, the prompt, is very narrow. Yes, I just did a very crude experiment.

pishpash 2y ago

Internal vectors aren't in any particular language.

daemonologist 2y ago
No, but theoretically, blunt responses might be more common in Dutch-language training data. A well-fit model would be expected to replicate that. (Basically similar to straight up asking it to be more blunt, except it has probably trained a lot more on _Dutch_ than on _someone just told me to be blunt_ so the effect might be more natural and more subtle.)

daemonologist 2y ago

I wonder (if this works at all) if the effect might be stronger if you also prompted in Dutch, preferably written by a fluent speaker rather than machine-translated.