11 ms·
Fluent 1.0: a localization system for natural-sounding translations
- indentit 8y agoit'd be great to see syntax highlighting for FTL files in the popular text editors, but I guess they would have to be unofficial unless a member of Mozilla/Fluent team wants to maintain it...
- indentit 8y agoI see that the playground has syntax highlighting and uses `ace`, whose syntax definitions are defined in JavaScript [0] and look fairly usable. I guess it could even be converted into a `.sublime-syntax` file without too much trouble :) [0]: https://github.com/projectfluent/play/blob/a4f49a4a7eeb93535a6243be81c7c2a2b553c904/client/editor-mode-fluent.js#L19-L161 https://github.com/projectfluent/play/blob/a4f49a4a7eeb93535...
- zbraniecki 8y agoI started https://github.com/projectfluent/fluent.vim https://github.com/projectfluent/fluent.vim but my vimscript is terrible and I will take all help I can get :)
- stasm 8y agoYou can get decent highlighting for basic Fluent messages by setting the editor to a mode for Properties files. For instance, whenever I type Fluent examples in GitHub, I use the following markdown: ```properties # A comment hello = Hello, world! ``` This is in fact by design. Properties files are quite nice for simple things. Fluent builds on top of them, and provides modern features like multiline text blocks (as long as it's indented, it's considered text continuation) and the micro-syntax for expressions: {$var}, {$var -> ...} etc. So far, I haven't had much time to invest in building proper highlighting modes for popular editors. There are ACE and Vim modes mentioned in other comments here, and also a slightly outdated https://atom.io/packages/language-ftl20n https://atom.io/packages/language-ftl20n written by a contributor. I'd love to see more such contributions, and I'll be more than happy to help by reviewing code!
- pcr910303 8y agoOMG this is so cool to a person who lives in the CJK world (to be specific, I’m Korean) where the order of noun/verb/adj is reversed and always gets to see programs that display text something like ‘Site is news reader HN’, ‘Button press confirm to’. It’s a pity that the programming world is still super bad at i18n :-(
- chungleong 8y agoIt won't help much with Chinese localization, where the persistent problem is that developers assume every language has the words "yes" and "no".
- themacguffinman 8y agoDoesn't Chinese have understandable localizations for affirmative/negative responses like "是" and "没有"? I don't see the problem.
- chungleong 8y agoThat's roughly like putting [OK] and [Cancel] into a dialog box that asks "Do you want to save this document?". Perfectly understandable but awkward, somewhat confusing to non-tech people--in a word, unfluent. Answering such questions in Chinese (and I imagine, in Welsh) requires the verb that was used in the question.
- zbraniecki 8y agoHi! This is also a problem in Welsh [0]. In the Fluent world we solve that on the level of design principles (do not reuse the string `yes` because it can be translated to a different `yes` in different buttons) and bindings (compound messages allow us to localize a whole widget with a main message and its yes/no options). [0] http://psychopixi.com/languages/yes-and-no/ http://psychopixi.com/languages/yes-and-no/
- zoul 8y agoI18n is something like security, it takes a lot of effort to get right, people often don’t know that their assumptions are subtly wrong, and getting them right often requires using a more complicated solution. No wonder programmers just cop out and settle for ASCII, naïve string comparisons, first name & surname fields, etc. And the problems go quite deep, all the way down to standard libraries and programming languages. The Swift debate about correct and performant Unicode string processing was very interesting – people are very hard to convince that string is not a collection of characters randomly accessible with integer indexes.
- fxfan 8y agoI joke that prima face fluent is to gettext/other i18n frameworks that rust is to c/c++ , well designed and thoughtful.
- theon144 8y agoNice! This honestly sounds really great; coincidentally as a Czech person, the multiple plural forms have sort of been a bane of i18n for me, a surprising amount of solutions don't even take this into account at all (?!) or require dumb workarounds. With Mozilla's experience and adherence to ideals of interoperability and openness, I can see Fluent as a solid "golden standard" solution for a great chunk of i18n needs :)
- zoul 8y agoAhoj! :) This is a perfect article that always comes to mind when talking pluralization: https://metacpan.org/pod/distribution/Locale-Maketext/lib/Locale/Maketext/TPJ13.pod https://metacpan.org/pod/distribution/Locale-Maketext/lib/Lo... We Czechs have it easy!
- Tomte 8y agoGreat article! I'm always impressed by the thought going into Perl libraries. There was a presentation years ago, how Perl handled Unicode right and every other programming language didn't (with Python 3 pretty close, IIRC). Does anyone remember the URL?
- arnsholt 8y agoI think the presentation you're thinking of is Tom Christiansen's Unicode: The Good, the Bad, and the (mostly) Ugly: https://www.azabani.com/pages/gbu/ https://www.azabani.com/pages/gbu/
- Tomte 8y agoI think that's the one. Thank you!
- lizmat 8y agoIf it's about Perl 6, perhaps: http://jnthn.net/papers/2015-spw-nfg.pdf http://jnthn.net/papers/2015-spw-nfg.pdf ?
- quelltext 8y agoIn the Czech example, how is it obvious that `few` stands for 2, 3, 4? Is that just how the concept of "few" (note the English term) is defined and understood by all Cezch speakers and thus this language specific meaning is encoded by Mozilla to map to the range 2-4? My point is that while there might be a concept of "few" that does map uniquely to that range, I am not sure naming the keyword "few" is the right name for this. Quite honestly it would be easier to understand if it were explicitly referring to the range. After all these strings are provided specific to a language anyway. As such why not encode the rules in them explicitly instead of relying on keywords? Or are those merely user defined abstractions to accomplish reuse? I guess it would help, but I'm still not sure why this needs a whole new framework.
- stasm 8y ago(Author of the blog post here.) Great question, thanks! Unicode defines six categories of plural forms: zero, one, two, few, many, and other. The names of these categories always appear in English. Unicode also maintains a collection of all mappings of numerical rules to these categories, for all languages supported by the CLDR. See http://www.unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html#cs http://www.unicode.org/cldr/charts/latest/supplemental/langu... for the mapping corresponding to the Czech grammar.
- quelltext 8y agoAh, if this is building on top of existing rules / standards that makes total sense.
- Zenbit_UX 8y agoCare to comment about how it might handle one bold word or a link mid sentence?
- stasm 8y agoWhen Fluent formats translations, it returns simple strings (in the sense of primitive computer types). They can include markup which is parsed by a higher-level abstraction responsible for actually showing the translations somewhere in the UI. Take a look at https://github.com/projectfluent/fluent.js/wiki/DOM-Overlays https://github.com/projectfluent/fluent.js/wiki/DOM-Overlays in the experimental fluent-dom package, and their React equivalent, https://github.com/projectfluent/fluent.js/wiki/React-Overlays https://github.com/projectfluent/fluent.js/wiki/React-Overla....
- m12k 8y agoThis look really good! As someone in ruby-land I wish there was some easy way to get notified if/when a fluent-ruby gem becomes available.
- arendtio 8y agoSounds more like a programming language for translations (than like a message format).
- lewisl9029 8y agoOne thing that I've always struggled w.r.t i18n is having to split messages up into often times less coherent chunks in order to add things like links or tooltips or styling elements in the middle of the text, which makes it more difficult to localize messages as a holistic piece independent of the source language. For a slightly contrived example to demonstrate this, let's say you have a string like this: "Please click here 7 times to confirm" Where you want to make the "click here 7 times" look like a link by wrapping it in a <a> tag, or just styled differently using a styled <span>. Using something like react-intl, which is what I've used in the past, you'd have to do something like this: <FormattedMessage id="confirm" defaultMessage={`Please { confirmLink } to confirm`} values={{ confirmLink: <a> <FormattedMessage id="confirm-link" defaultMessage={`click here {clickCount, number} {clickCount, plural, one {time} other {times} }`} values={{ clickCount: 7 }} /> </a> }} /> If then some language happens to require a completely different sentence structure that changes the ordering such that the "to confirm" part needs to be interleaved somewhere in the middle of the "click here 7 times" message to sound fluent, this would not be able to accommodate that. I'm wondering how people generally deal with this, in React and elsewhere.
- Sylos 8y agoFrom an accessibility point of view, it's also recommended to avoid links that only span over such half-sentences. Screen reader users will often navigate your page by cycling through the links that are on the page and then they'll get only the link-text read out, not the surrounding text.
- lewisl9029 8y agoMore and more, I'm starting to think maybe plain text isn't always the best abstraction to be using for defining messages for i18n. If messages were to be defined in terms of whatever primitive you're using to build your UI (i.e. React components if you're using React, and raw html template nodes if you're working with plain html), then all of this impedance mismatch might disappear. In the React case, a component oriented approach to i18n could maybe look something like this: const DefaultMessage = ({ clickCount }) => <span> Please <a>click here {clickCount} {pluralize(clickCount, {one: "time", other: "times"})} </a> to confirm </span> // some theoretical language that requires putting "to confirm" between // "click here" and "7 times", using English for clarity const MessageInSomeOtherLanguage = ({ clickCount }) => <span> Please <a>click here to confirm {clickCount} {pluralize(clickCount, {one: "time", other: "times"})} </a> </span> // some theoretical component that renders different components // based on the language and passes through props <FormattedComponent id="confirm" defaultMessage={DefaultMessage} props={{ clickCount: 7 }} /> This feels a lot more elegant and flexible to me. Though it would make it more difficult for non-technical folks to contribute to translations, which might not be much of a concern if your company has the resources to support localization teams in-house. Am I overlooking any other obvious downsides to this approach? Does anyone know of any libraries that offers a similar API, or have experience using a similar approach?
- ktpsns 8y agoIsn't it amazing that even with these "apparently solved and very basic" problems like i18n, there are still so many low-hanging fruits, and an open source project can do better than many companies. I'm German and I disabled spell checking almost everywhere, because most implementations are extremely poor in German. Word lists are a poor solution to capture different word forms, and I find it surprising that even in 2019, only very few programs get that right (for instance Microsoft Word; it understands some but not all grammar rules). This is another thing where I think a modern (OSS) spell checker could make a difference.
- pluma 8y agoI guess my biggest pet peeve with German spellcheckers and autocomplete solutions other than the nonexistent support for compound words is that most of them don't understand capitalisation rules.
- wongarsu 8y agoNot just spellcheckers, virtual keyboards too. Writing German on a smartphone can be slightly infuriating
- ngrilly 8y agoThe comparison with gettext is really interesting: https://github.com/projectfluent/fluent/wiki/Fluent-vs-gettext https://github.com/projectfluent/fluent/wiki/Fluent-vs-gette... Especially the advantages and drawbacks of using the source string as a message identifier, compared to a developer provided ID. I'm wondering if fluent has something similar to xgettext, to extract the IDs from the source code? Edit: Looks like there is some discussion about extraction here: https://github.com/projectfluent/fluent.js/wiki/React-Bindings-Discussion https://github.com/projectfluent/fluent.js/wiki/React-Bindin...
- stasm 8y agoThere's a tool written by a Udacity developer which can be used to autoextract copy from React source code: https://github.com/udacity/fluent-react-utils/tree/master/packages/extraction https://github.com/udacity/fluent-react-utils/tree/master/pa...
- ngrilly 8y agoI was thinking of something along this line. Thanks!
- olau 8y agoDon't take the comparison at its face value, it's clear to me that whoever wrote it isn't really familiar with gettext, or deliberately talking it down. Yes, it's sort of ancient, but the problems mentioned can be solved. And using the source string as ID is a pretty clever trick. Of course, there are some downsides, but there are certainly also downsides with separate IDs. Having said that, Fluent looks interesting.
- zbraniecki 8y agoHi! Thank you for your feedback. I'm one of the authors of Fluent, and this wiki article. I have, in fact, been using Gettext for quite a few years, but of course, as you pointed out, I am also biased. If you have suggestions on how to improve the article to better represent the reality, please, file an issue and provide a PR! Our goal is to express our design differences, but we don't want to mislead anyone!
- Vinnl 8y agoHanding control of inflections etc. over to the translator rather than the developer is one of those great ideas that make so much sense when you first see them, that you start to wonder why we didn't do this before. Great work by Mozilla; it's clear there's a lot of experience in the organisation feeding into the design of this system, and it's great that they're sharing it with the world.
- ajuc 8y agoDoes it handle the "x of y" in Slavic languages correctly? For example in Polish: "Page 3 of 4" is "Strona 3 z 4" "Page 3 of 100" is "Strona 3 ze 100"
- lifthrasiir 8y agoAs far as I can tell Fluent has no builtin support for tagging a phonemic information to messages or arguments (it only supports plural rules and number formatting largely derived from the CLDR). You can probably specify special cases with selectors (but it will quickly go absurd, for Polish I guess that applies to 6, 7, 16, 17, 60..79, 100..199, 600..699 and so on?) or have an external function. Context: Polish preposition "z(e)" is spelled "z" if the following word starts with s and z and alikes (problematically enough, the exact rule is not systematic) and "ze" otherwise. Korean has a similar case with postpositions "은(는)" and "이(가)" where the former is for words ending with a consonant and the latter is for words ending with a vowel, and the ko-KR localization of Firefox seems to completely ignore and/or sidestep this; the last letter is assumed (e.g. "{-brand-name}는") or a static word is inserted (e.g. "{$user} 사용자는" instead of "{$user}은(는)").
- ajuc 8y agoFunnily enough "Strona 3 z 6" and "Strona 3 z 7" sounds correct but "Strona 3 z 100" doesn't. So I think it's only words starting with "s", not "sz" nor "si". And only 0 starts with "z" so it's not a problem (you never have "X out of 0"). So the only special case is for 100-199, 100 000-199 999, etc.
- lifthrasiir 8y agoYeah, it is clear that I don't speak Polish ;-) What is a common workaround there? "Strona X z(e) Y"?
- ajuc 8y ago> What is a common workaround there? Ignoring the issue altogether :) Or, if you're pedantic - implementing the special cases in the source code. But that's unmaintenable if you have lots of languages.
- progx 8y agoHope someday somebody invent .pot files, plurals and gettext.
- pluma 8y agoNo need to be sarcastic. The documentation explains the advantages over gettext. Most notable being that using the source language string as identifier a) discourages changes (and improvements) to the source language strings and b) makes it hard to handle strings that appear the same in the source language but need different translations.
- santialbo 8y agoLooking at the examples, it looks like something like messageformat (https://messageformat.github.io/messageformat/ https://messageformat.github.io/messageformat/) would have been a good solution to them. We've been using this at my job and we are very happy with the flexibility it provides. The hard part comes when a third party has to do the translations because someone from tech needs to be involved.
- Schoolmeister 8y agoFrom the article: "Many key ideas in Fluent have also been inspired by XLIFF and ICU’s MessageFormat."
- santialbo 8y agoThank you for pointing out. I found a comparison wiki article. I think this is an improvement over messageformat. They have tackled many of the issues with mf I didn't even know I had :P
- okuryu 8y agoWhat's advantage over the FormatJS suite? https://formatjs.io/ https://formatjs.io/
- zbraniecki 8y agoHi! FormatJS is very similar to some of our bindings, and is powered by MessageFormat on the lower level. Here's our take on the differences between MessageFormat and Fluent - https://github.com/projectfluent/fluent/wiki/Fluent-and-ICU-MessageFormat https://github.com/projectfluent/fluent/wiki/Fluent-and-ICU-...
- megous 8y agoRe: genedered pluralized example in the article. How will the system deal with the fact, that in some languages (Czech, too): ($count -> Jana added {n} {apples|apple}) ($gender -> to {his|her} profile) the $gender will affect what form the word "added" should take. You're suddenly dealing with possibilities($count)*possibilities($gender) variants of the sentence
- tobr 8y agoWouldn’t you just wrap that word in another selector?
- megous 8y agoWill the system still be sanely usable for translators?
- 60654 8y agoYeah this is where these things fall apart. I haven't used Mozilla's Fluent but I used a very similar closed source system at another company some years back. Some failure modes: - Gender agreement is not trivial. In French, in "Mary bought it" the verb needs to match the gender and number of the object not the subject: "Marie l'a acheté" vs "Marie l'a achetée" vs "Marie les a achetés" depending on the gender/number of the "it" object. But in most other cases the verb needs to match the subject in gender and number, in Polish "Maria kupila" vs "Stas kupil" vs "Oni kupili". - In many languages nouns need to agree in case, gender, and number with the phrase they're in, even in English we see this with pronouns: "this is he" vs "this is his". - And not to mention number agreement between pronoun and either subject or object, depending on context: "this is the button" vs "these are the buttons" - but also "this hovers over the button" vs "these hover over the button" etc. Pronouns in general are a world of hurt, as are copulae (is/are/etc) - So once we want more complex sentences, simple word tagging like $gender becomes insufficient, because now there's multi-party agreement to worry about, we have to worry about $gender_subject and $object_gender_number_case, etc. This becomes completely untenable for all but the most technical translators. Maybe those are easy to find for a world famous project like Firefox. Unfortunately, not so for a run of the mill commercial project.
- revskill 8y agoThis is how to pass variables to localized React Component. The API is elegant. Bravo to the team ! https://github.com/projectfluent/fluent.js/blob/master/fluent-react/examples/text-input/src/App.js#L29 https://github.com/projectfluent/fluent.js/blob/master/fluen...
- revetkn 8y agoMy take on how to solve the natural-sounding translation problem: https://www.lokalized.com/#a-more-complex-example https://www.lokalized.com/#a-more-complex-example The magic is a tiny expression language which understands plural cardinalities, ordinals, etc. so a translator can encode all required logic in a JSON file - the application code can be "dumb".