5 ms·

Sorry Polish is not one of the supported language yet: Arabic; Chinese (Simplified); Chinese (Traditional); Cyrillic; Danish; Dutch; English; Farsi; Finnish; Fr

by pesenti 12y ago

Sorry Polish is not one of the supported language yet: Arabic; Chinese (Simplified); Chinese (Traditional); Cyrillic; Danish; Dutch; English; Farsi; Finnish; French; German; Greek; Hebrew; Hindi; Icelandic; Italian; Japanese; Korean; Norwegian (Bokmal); Norwegian (Nynorsk); Portuguese; Spanish; Swedish; Turkish; Urdu.

TeMPOraL 12y ago
Oh right, I'm sorry. It was actually written on the demo page, but I somehow missed it.
johnward 12y ago
I also imagine that distinguishing between many of these languages is not a trivial task for a computer to handle.
- TillE 12y ago
  You'd get collisions if you're trying to determine the language of a single word, obviously. But a sentence or two should be no problem. You're just matching words in a dictionary. Different languages are mostly quite distinct. It would be a more interesting challenge to recognize and understand, say, German dialects.
  - a3_nm 12y ago
    In fact you can already get very good results, even without a dictionary, just with a distribution of the trigrams in your input languages (which is immediate to compute from a corpus). Just compute the distribution of the sample and return the language that has the closest one.
    - tripzilch 12y ago
      Even a single letter frequency histogram is already a very powerful discriminator to detect language. Although I suppose scoring maximum likelihood on bi- or trigrams works better for very small (sub-sentence) bits of text.
wtbob 12y ago
Odd that Farsi is supported, not Polish, given that Poland's GDP is close to Iran's and that Poland has no sanctions against it. Perhaps IBM is planning to sell lots of Watson-powered systems in Afghanistan & Tajikistan? Incidentally, Poland's GDP is several times that of Finland…
- pridkett 12y ago
  Polish as a language is more similar to other languages than Finnish or Farsi. Basically, it's easier to detect Farsi and Finnish because they're more distinct and in smaller language families than Polish, which is similar to many slavic languages.
- jameshart 12y ago
  Do you not think there might be a market for software for analyzing text in Farsi perhaps -outside- the region where it's actually spoken? (disclaimer - work for IBM, but not on Watson, or anything related to intelligence analytics)
angrypoulpe 12y ago
French is identified as EN_us -> http://imgur.com/VdDJVw4 http://imgur.com/VdDJVw4 Inadmissible !
- parsabg 12y ago
  AYLIEN's Text API does a better job :-) http://aylien.com/text-api-demo?text=vache%20cochon%20metal%20jolie%20coffre%20a%20poil%20m%C3%A9chant%20solitaire%20ubuesque%20qualit%C3%A9%20corridor%20chausettes&tab=language&run=1 http://aylien.com/text-api-demo?text=vache%20cochon%20metal%... (disclaimer: I'm the founder of AYLIEN)

by pesenti 12y ago

TeMPOraL 12y ago

Oh right, I'm sorry. It was actually written on the demo page, but I somehow missed it.

johnward 12y ago

I also imagine that distinguishing between many of these languages is not a trivial task for a computer to handle.

TillE 12y ago
You'd get collisions if you're trying to determine the language of a single word, obviously. But a sentence or two should be no problem. You're just matching words in a dictionary. Different languages are mostly quite distinct. It would be a more interesting challenge to recognize and understand, say, German dialects.
- a3_nm 12y ago
  In fact you can already get very good results, even without a dictionary, just with a distribution of the trigrams in your input languages (which is immediate to compute from a corpus). Just compute the distribution of the sample and return the language that has the closest one.
  - tripzilch 12y ago
    Even a single letter frequency histogram is already a very powerful discriminator to detect language. Although I suppose scoring maximum likelihood on bi- or trigrams works better for very small (sub-sentence) bits of text.

wtbob 12y ago

Odd that Farsi is supported, not Polish, given that Poland's GDP is close to Iran's and that Poland has no sanctions against it. Perhaps IBM is planning to sell lots of Watson-powered systems in Afghanistan & Tajikistan? Incidentally, Poland's GDP is several times that of Finland…

pridkett 12y ago
Polish as a language is more similar to other languages than Finnish or Farsi. Basically, it's easier to detect Farsi and Finnish because they're more distinct and in smaller language families than Polish, which is similar to many slavic languages.
jameshart 12y ago
Do you not think there might be a market for software for analyzing text in Farsi perhaps -outside- the region where it's actually spoken? (disclaimer - work for IBM, but not on Watson, or anything related to intelligence analytics)

angrypoulpe 12y ago

French is identified as EN_us -> http://imgur.com/VdDJVw4 http://imgur.com/VdDJVw4 Inadmissible !

parsabg 12y ago
AYLIEN's Text API does a better job :-) http://aylien.com/text-api-demo?text=vache%20cochon%20metal%20jolie%20coffre%20a%20poil%20m%C3%A9chant%20solitaire%20ubuesque%20qualit%C3%A9%20corridor%20chausettes&tab=language&run=1 http://aylien.com/text-api-demo?text=vache%20cochon%20metal%... (disclaimer: I'm the founder of AYLIEN)