6 ms·

though useless for many non english languages

by wahnfrieden 2y ago

keithalewis 2y ago
[flagged]
- zarathustreal 2y ago
  My brother in Christ, releasing a tool for free does not exempt it from criticism
epcoa 2y ago
Eh? The tokenizers are fully pluggable, the Unicode and ICU tokenizers will be good for many purposes and you can always drop something specific in. Eg https://github.com/wangfenjin/simple https://github.com/wangfenjin/simple
- wahnfrieden 2y ago
  I know it’s extensible, with original work. ICU isn’t helpful for some languages
  - AlexeyBelov 2y ago
    For what languages? Why ICU isn't helpful? My naive guess would be "it works for anything supported by Unicode".
    - wahnfrieden 2y ago
      languages that don't use spaces like Japanese
troupo 2y ago
Of all/many opensource projects I looked at, only meilisearch is good for non-English languages: https://www.meilisearch.com/docs/learn/resources/language https://www.meilisearch.com/docs/learn/resources/language
- wahnfrieden 2y ago
  Japanese support looks good. But it looks like I can’t deploy this inside an iOS app :(
  - plq 2y ago
    Why not?
    - wahnfrieden 2y ago
      I need an offline client side db/ fts (for this language learning app: https://reader.manabi.io https://reader.manabi.io )
      - plq 2y ago
        To my knowledge, meili is free software (MIT license) so theoretically, it should be possible to embed it to an IOS app.
        wahnfrieden 2y ago
        it's more about the tech it uses... it's oriented toward docker based deploy and is made in rust so maybe it can be ported to iOS, I'd have to spend time forking the packaging. looks like it's geared toward server environments
- djhn 2y ago
  Unfortunately it still works very poorly for languages with compound words (German, Swedish, Finnish) where you would usually want to surface search results if someone uses the second or third part of the compound word, otherwise the user has to enumerate the (often too many) variants. Meilisearch doesn’t support suffix search and their tokenizer can’t handle these languages. For instance searching for ‘___missilestrike’ in Kyiv would ideally surface all incidents, regardless of launch mode (surface-to-surface, air-to-surface…), range, propulsion, warhead, or guidance system. In these languages a number of these would form the prefix of that compound word.
deleted 2y ago
[deleted]

though useless for many non english languages

keithalewis 2y ago

[flagged]

zarathustreal 2y ago
My brother in Christ, releasing a tool for free does not exempt it from criticism

epcoa 2y ago

Eh? The tokenizers are fully pluggable, the Unicode and ICU tokenizers will be good for many purposes and you can always drop something specific in. Eg https://github.com/wangfenjin/simple https://github.com/wangfenjin/simple

wahnfrieden 2y ago
I know it’s extensible, with original work. ICU isn’t helpful for some languages
- AlexeyBelov 2y ago
  For what languages? Why ICU isn't helpful? My naive guess would be "it works for anything supported by Unicode".
  - wahnfrieden 2y ago
    languages that don't use spaces like Japanese

troupo 2y ago

Of all/many opensource projects I looked at, only meilisearch is good for non-English languages: https://www.meilisearch.com/docs/learn/resources/language https://www.meilisearch.com/docs/learn/resources/language

wahnfrieden 2y ago
Japanese support looks good. But it looks like I can’t deploy this inside an iOS app :(
- plq 2y ago
  Why not?
  - wahnfrieden 2y ago
    I need an offline client side db/ fts (for this language learning app: https://reader.manabi.io https://reader.manabi.io )
    - plq 2y ago
      To my knowledge, meili is free software (MIT license) so theoretically, it should be possible to embed it to an IOS app.
      - wahnfrieden 2y ago
        it's more about the tech it uses... it's oriented toward docker based deploy and is made in rust so maybe it can be ported to iOS, I'd have to spend time forking the packaging. looks like it's geared toward server environments
djhn 2y ago
Unfortunately it still works very poorly for languages with compound words (German, Swedish, Finnish) where you would usually want to surface search results if someone uses the second or third part of the compound word, otherwise the user has to enumerate the (often too many) variants. Meilisearch doesn’t support suffix search and their tokenizer can’t handle these languages. For instance searching for ‘___missilestrike’ in Kyiv would ideally surface all incidents, regardless of launch mode (surface-to-surface, air-to-surface…), range, propulsion, warhead, or guidance system. In these languages a number of these would form the prefix of that compound word.

deleted 2y ago

[deleted]