6 ms·

There's a fun way to test translation systems. Have it translate something to another language, and then translate it back (after starting a new chat/clearing i

There's a fun way to test translation systems. Have it translate something to another language, and then translate it back (after starting a new chat/clearing its memory) and compare. It's best done with some sampling of text that has style, metaphor, parable, etc. After all I think we should be more adversarial than promotional when testing such things. Here [1] is the first paragraph of a human translation of Hantsel and Gretel: --- "Hard by a great forest dwelt a poor wood-cutter with his wife and his two children. The boy was called Hansel and the girl Gretel. He had little to bite and to break, and once when great dearth fell on the land, he could no longer procure even daily bread. Now when he thought over this by night in his bed, and tossed about in his anxiety, he groaned and said to his wife: 'What is to become of us? How are we to feed our poor children, when we no longer have anything even for ourselves?' 'I'll tell you what, husband,' answered the woman, 'early tomorrow morning we will take the children out into the forest to where it is the thickest; there we will light a fire for them, and give each of them one more piece of bread, and then we will go to our work and leave them alone. They will not find the way home again, and we shall be rid of them.' 'No, wife,' said the man, 'I will not do that; how can I bear to leave my children alone in the forest?--the wild animals would soon come and tear them to pieces.' 'O, you fool!' said she, 'then we must all four die of hunger, you may as well plane the planks for our coffins,' and she left him no peace until he consented. 'But I feel very sorry for the poor children, all the same,' said the man." --- And here is GPT 3.5's final rendition of it, after converting it to Japanese and back: --- "Near a large forest, a poor woodcutter lived with his wife and two children. The boy was named Hansel, and the girl was Gretel. They had little to eat, and during a severe famine in the country, they could no longer even provide daily meals. One night, as he lay in bed pondering this, the woodcutter, filled with anxiety, said to his wife, "What will become of us? How will we feed our poor children when we no longer have anything for ourselves?" His wife replied, "Husband, tomorrow morning, let's take the children into the forest. There, we will light a fire and give each of them one more piece of bread. Then, we will go to work and leave the children on their own. They won't be able to find their way back, and we will be freed from them." The man said, "No, wife, I cannot do that. How can I leave my own children alone in the forest? Wild animals will come and tear them apart." She retorted, "You fool! In that case, we will all four perish of hunger, and you can just as well plane the boards for our coffins," and she didn't leave him in peace until he agreed. "But the poor children are so pitiful," the man said." --- It's a generally terrible translation with a simple (though heavily stylized) text. Not only is absolutely all style, imagery, and character eliminated from the writing, but it's also overtly nonsensical in parts. Lighting a fire for the children ends up translated as simply lighting a fire, as if the goal is to start a forest fire. Similarly the man's sorrow for the children ends up being completely lost, and is translated as him simply calling his children pitiful. I'd be curious to see GPT4's output on this, or ideally even more challenging paragraphs. [1] - http://www.literaturepage.com/read/grimms-fairy-tales-72.html http://www.literaturepage.com/read/grimms-fairy-tales-72.htm...

totetsu 3y ago
Maybe Japanese is not a good target language for this type of test, because its very.. high context, that is to say. the natural way of expressing the ideas of the origional text in japanese, might leave some of the meaning to context, and not have it explicitly marked or encoded in words left on the page, then when that is translated back to english that part is lost. also it doesnt provide a good test of if the japanese in the middle was any good.. like in this case it probably had a lot of "watashi"s in place of "I" where this woud not be natural in japanese. The fun part of chatgpt is you can ask it to do non-standard translations like "can you give me the direct non idiomatic translation of the following Japanese into English, to help me remember these phrases in English words but the Japanese way of saying it" Sure, I can provide a direct, non-idiomatic translation of the Japanese text into English. This kind of task is straightforward for me, so no worries. "Big forest's right next to, poor wood-cutter and his wife, and two children were living. Boy was Hansel, girl was Gretel called. He had almost no food, once big famine attacked the country, daily bread even could no longer obtain. Night, in bed this thing thinking, anxious tossing and turning, he groaned, to wife said: 'We what become? Poor children to how food provide, ourselves even nothing when.' 'Husband, tomorrow morning early, children to forest's thickest place take let's go. There for them fire light, each one more piece of bread fragment give, then we to work go, them alone leave. They home return way find not become, we them let go can.' 'No, wife,' man said, 'that can't do, why I children in forest alone leave can? Wild animals soon come, them tear apart will.' 'Oh, fool!' she said, 'then we four people all die of hunger, our coffins' plank plane might as well.' And, she him agree until him peace not did. 'But, still poor children very sorry,' man said."
pohuing 3y ago
Asking it to translate text that most certainly was in the training data even seems like a bit of a cheat doesn't it?
BasedAnon 3y ago
This is not a good test of translation systems. It's like pointing out that compressing a zip file lead to a larger file, or converting a JPG to a PNG and then saving it as a JPG again lead to more artifacting.

totetsu 3y ago

Maybe Japanese is not a good target language for this type of test, because its very.. high context, that is to say. the natural way of expressing the ideas of the origional text in japanese, might leave some of the meaning to context, and not have it explicitly marked or encoded in words left on the page, then when that is translated back to english that part is lost. also it doesnt provide a good test of if the japanese in the middle was any good.. like in this case it probably had a lot of "watashi"s in place of "I" where this woud not be natural in japanese. The fun part of chatgpt is you can ask it to do non-standard translations like "can you give me the direct non idiomatic translation of the following Japanese into English, to help me remember these phrases in English words but the Japanese way of saying it" Sure, I can provide a direct, non-idiomatic translation of the Japanese text into English. This kind of task is straightforward for me, so no worries. "Big forest's right next to, poor wood-cutter and his wife, and two children were living. Boy was Hansel, girl was Gretel called. He had almost no food, once big famine attacked the country, daily bread even could no longer obtain. Night, in bed this thing thinking, anxious tossing and turning, he groaned, to wife said: 'We what become? Poor children to how food provide, ourselves even nothing when.' 'Husband, tomorrow morning early, children to forest's thickest place take let's go. There for them fire light, each one more piece of bread fragment give, then we to work go, them alone leave. They home return way find not become, we them let go can.' 'No, wife,' man said, 'that can't do, why I children in forest alone leave can? Wild animals soon come, them tear apart will.' 'Oh, fool!' she said, 'then we four people all die of hunger, our coffins' plank plane might as well.' And, she him agree until him peace not did. 'But, still poor children very sorry,' man said."

pohuing 3y ago

Asking it to translate text that most certainly was in the training data even seems like a bit of a cheat doesn't it?

BasedAnon 3y ago

This is not a good test of translation systems. It's like pointing out that compressing a zip file lead to a larger file, or converting a JPG to a PNG and then saving it as a JPG again lead to more artifacting.