5 ms·

Based on a 2-sec look at the code, it's using a built-in database of trigrams as a predictor of the language. https://github.com/wooorm/franc/blob/master/lib/d

by hywel 12y ago

Based on a 2-sec look at the code, it's using a built-in database of trigrams as a predictor of the language. https://github.com/wooorm/franc/blob/master/lib/data.json https://github.com/wooorm/franc/blob/master/lib/data.json

allan_s 12y ago
my bad, I've been looking to data folder first and haven’t found anything, I should have tried harder
riffraff 12y ago
the question would be where he got the language data If the original language data is available I'd suggest classifying the trigrams as "high" and "low" frequency, which should improve performance without needing to keep full frequency data.
- wooorm 12y ago
  No full-frequency data is kept, only 300 top-trigrams are identified. A quick through the source also reveals wooorm/trigrams, and wooorm/udhr, as sources!
  - riffraff 12y ago
    yes, I meant: keeping full frequency could have been avoided to save space/memory but having two classes high/low could be a good tradeoff.
    - wooorm 12y ago
      It’s an interesting thought. I might fiddle on it, but I’m not sure it would work in practice (d’oh). Thanks!