2 ms·
> https://github.com/uav4geo/LibreTranslate/issues/33 https://github.com/uav4geo/LibreTranslate/issues/33 That issue is about emojis, which confused me a bit,
by yorwba 6y ago
> https://github.com/uav4geo/LibreTranslate/issues/33 https://github.com/uav4geo/LibreTranslate/issues/33
That issue is about emojis, which confused me a bit, but I see that the linked commit is about replacing <unk> by the corresponding source token. https://github.com/argosopentech/argos-translate/commit/6a0f389caccd35645049f00658636b5c007b98a8 https://github.com/argosopentech/argos-translate/commit/6a0f...
That's definitely an improvement, but what I was actually wondering about was why the model was copying the input verbatim in the first place.
Looking through the en-zh data in Opus, it looks like a bit of a mixed bag. This sample from OpenSubtitles v1 contains an ad for MyDVDrip in the Chinese version only: http://opus.nlpl.eu/OpenSubtitles/v1/en-zh_sample.html http://opus.nlpl.eu/OpenSubtitles/v1/en-zh_sample.html Filtering the data with some heuristics might be a good idea.
- pjfin123 6y agoYou're right there's surprisingly little data available for English-Chinese. I'm not sure why, it seems like there would be a lot of demand for translating between them. For the en-zh model copying this is a known issue: https://github.com/argosopentech/argos-translate/issues/4 https://github.com/argosopentech/argos-translate/issues/4
- yorwba 6y agoThere's a lot of demand, but the people most likely to satisfy that demand for free are Chinese fan translators, and they're more likely to upload their work to Chinese sites where Western dataset collectors are unlikely to find them...