3 ms·

I’ve wondered about this more generally (ie, simply prompting in different languages). For example, if I ask for a pasta recipe in Italian, will I get a more a

I’ve wondered about this more generally (ie, simply prompting in different languages). For example, if I ask for a pasta recipe in Italian, will I get a more authentic recipe than in English? I’m curious if anyone has done much experimenting with this concept. Edit: I looked up Sapir-Whorf after writing. That’s not exactly where my theory started. I’m thinking more about vector embedding. I.e., the same content in different languages will end up with slightly different positions in vector space. How significantly might that influence the generated response?

astrange 9mo ago
The answer is yes, LLMs have different behavior and factual retrieval in different languages. I had some papers about this open earlier today but closed them so now I can't link them ;(
marssaxman 9mo ago
I just tried your experiment, first asking for a bolognese sauce recipe in English, then translating the prompt to Italian and asking again. The recipes did contain some notable differences. Where the English version called for ground beef, the Italian version used a 2:1 mix of beef and pancetta; the Italian version further recommended twice as much wine, half as much crushed tomato, and no tomato paste. The cooking instructions were almost the same, save for twice as long a simmer in the Italian version. More authentic, who knows? That's a tricky concept. I do think I'd like to try this robot-Italian recipe next time I make bolognese, though; the difference might be interesting.
- rkomorn 9mo ago
  FWIW, and tangential, the biggest (and time consuming) difference I ever found in making bolognese was hand cutting the meat instead of getting it ground. The texture was way better. It's a pain to do (obviously) but worth trying at least once, IMO.
  - marssaxman 9mo ago
    Thanks for the recommendation. Diced pancetta is readily available here, but I'd have to chop up the beef myself; which cut did you use?
    - rkomorn 9mo ago
      Recipe calls for skirt steak or chuck. I used chuck. Skirt steak would probably taste nicer, though, but might also be harder to chop. I ended up chopping it down to 2-3mm (~1/8in?) bits, and it helps to have the meat really cold (eg having hung out in the freezer for a bit).
- silvanocerza 9mo ago
  The italian counterpart of what english speakers call "bolognese sauce" would be "ragù alla bolognese". I've never heard anyone call it "salsa bolognese", it's mostly called "ragù" only as it's most common type. Nonetheless ragù alla bolognese is made with ground beef and tomato sauce, so the italian version is simply wrong. Try and ask for ragù recipe instead. :)
  - marssaxman 9mo ago
    That is the phrase Google Translate proposed: the exact prompt I used was "Come si prepara il ragù alla bolognese?" I often consult several different versions of a recipe before cooking, and this feels like a normal degree of variation. Perhaps there are regional differences? Just for kicks, I asked (in English) "what is an authentic Italian recipe for bolognese ragu?", and it produced a recipe similar to the version returned from the Italian prompt, noting "This version follows the classic canon recognized by the Accademia Italiana della Cucina". Searching on name of that organization led me to this recipe: https://www.accademiaitalianadellacucina.it/sites/default/files/Rag%C3%B9%20alla%20bolognese%20-%20updated%20recipe_20%20April%202023.pdf https://www.accademiaitalianadellacucina.it/sites/default/fi...
    - silvanocerza 9mo ago
      The translation is right. There are indeed regional differences, but at that point is not called "alla bolognese" anymore but "alla whatever place". People usually call it "ragù" and that's it. Didn't know that the original recipe has pancetta too. It's good nonetheless. :)

astrange 9mo ago

The answer is yes, LLMs have different behavior and factual retrieval in different languages. I had some papers about this open earlier today but closed them so now I can't link them ;(

marssaxman 9mo ago

I just tried your experiment, first asking for a bolognese sauce recipe in English, then translating the prompt to Italian and asking again. The recipes did contain some notable differences. Where the English version called for ground beef, the Italian version used a 2:1 mix of beef and pancetta; the Italian version further recommended twice as much wine, half as much crushed tomato, and no tomato paste. The cooking instructions were almost the same, save for twice as long a simmer in the Italian version. More authentic, who knows? That's a tricky concept. I do think I'd like to try this robot-Italian recipe next time I make bolognese, though; the difference might be interesting.

rkomorn 9mo ago
FWIW, and tangential, the biggest (and time consuming) difference I ever found in making bolognese was hand cutting the meat instead of getting it ground. The texture was way better. It's a pain to do (obviously) but worth trying at least once, IMO.
- marssaxman 9mo ago
  Thanks for the recommendation. Diced pancetta is readily available here, but I'd have to chop up the beef myself; which cut did you use?
  - rkomorn 9mo ago
    Recipe calls for skirt steak or chuck. I used chuck. Skirt steak would probably taste nicer, though, but might also be harder to chop. I ended up chopping it down to 2-3mm (~1/8in?) bits, and it helps to have the meat really cold (eg having hung out in the freezer for a bit).
silvanocerza 9mo ago
The italian counterpart of what english speakers call "bolognese sauce" would be "ragù alla bolognese". I've never heard anyone call it "salsa bolognese", it's mostly called "ragù" only as it's most common type. Nonetheless ragù alla bolognese is made with ground beef and tomato sauce, so the italian version is simply wrong. Try and ask for ragù recipe instead. :)
- marssaxman 9mo ago
  That is the phrase Google Translate proposed: the exact prompt I used was "Come si prepara il ragù alla bolognese?" I often consult several different versions of a recipe before cooking, and this feels like a normal degree of variation. Perhaps there are regional differences? Just for kicks, I asked (in English) "what is an authentic Italian recipe for bolognese ragu?", and it produced a recipe similar to the version returned from the Italian prompt, noting "This version follows the classic canon recognized by the Accademia Italiana della Cucina". Searching on name of that organization led me to this recipe: https://www.accademiaitalianadellacucina.it/sites/default/files/Rag%C3%B9%20alla%20bolognese%20-%20updated%20recipe_20%20April%202023.pdf https://www.accademiaitalianadellacucina.it/sites/default/fi...
  - silvanocerza 9mo ago
    The translation is right. There are indeed regional differences, but at that point is not called "alla bolognese" anymore but "alla whatever place". People usually call it "ragù" and that's it. Didn't know that the original recipe has pancetta too. It's good nonetheless. :)