5 ms·

I'm not sure I'm close to solving it, but I have an approach that I'd like some feedback on. I have a corpus of text in many Indian languages, which i'd like t

I'm not sure I'm close to solving it, but I have an approach that I'd like some feedback on. I have a corpus of text in many Indian languages, which i'd like to index and search. The twist is that I'd like to support searches in English. The problem is that there are many phonetic transliterations of the same word (e.g the Hindi word for law can either be written as "qanoon" or "kanun"), and traditional spelling correction methods don't work because of excessive edit distance. My approach is this: Use some sequence to sequence ML technique (LSTM, GRU, ..., attention) to a query in English to the most probable translation and then use that to look it up using a standard document indexing toolkit like Lucene. (I can put together a training dataset of english transliterations of sentences to their original text) The problem is that I'd like the corpus, the index and the model to be all on a mobile. I have a suspicion that the above method won't straightforwardly fit on a mobile (for a few Gig of corpus text), and that the inference time may be long. Is this assumption wrong? How would you solve the problem? Would TinyML be a better approach for the inferencing part?

peripitea 5y ago
I'm not sure I understand the problem specification. You want to be able to search "law", and find documents containing "qanoon" or "kanun", right? How does your proposed solution handle that? It seems like the approach with ML TL -> Lucene would still only find one of the two, unless your model is written to return a set of possible transliterations. Or are you saying your approach doesn't currently solve this part of the problem, and that's one of the things you'd like input on? Is the corpus the only data you have, i.e. do you need to use it for training and validation as well? In terms of the size of the data, if you want to store the corpus on the phone anyway, won't the index and model be relatively small in comparison?
- sriram_malhar 5y ago
  No, sorry for the confusion. I want to be able to type “kanun” or “qanoon”, and have it infer the Hindi word “कानून”, which is an indexed word. It is not necessary that there is a one to one correspondence between words. Sometimes two english words may represent one hindi word, or vice versa. I believe I can build up a decent sized training/validation set, for example from Bollywood song lyric databases written in English, and mapping them to the Hindi equivalent (or Tamil, Bengali, etc). As for your last question, I dont know, since I haver implemented an ML model in practice. I saw a tutorial on Bert this morning, where a word has 768 features. That itself sounds huge, leave alone the model itself.
talolard 5y ago
A non ml way to approach this is to use phonetic distance, e.g. qanoon and kanun sound the same so they are close. There is an algorithm called Soundex with python implementations you can try.
- sriram_malhar 5y ago
  Refer what I said about edit distance. Traditional methods don't work well at all. The fundamental difference between my problem and traditional spelling correction algos is that in the latter, there is a canonical correct spelling to be used as a reference. In my problem, there isn't. There are different approximate ways of spelling out most hindi words ... there is no one correct way. There are common patterns, sure, but it is too tedious to encode all the variations.

peripitea 5y ago

I'm not sure I understand the problem specification. You want to be able to search "law", and find documents containing "qanoon" or "kanun", right? How does your proposed solution handle that? It seems like the approach with ML TL -> Lucene would still only find one of the two, unless your model is written to return a set of possible transliterations. Or are you saying your approach doesn't currently solve this part of the problem, and that's one of the things you'd like input on? Is the corpus the only data you have, i.e. do you need to use it for training and validation as well? In terms of the size of the data, if you want to store the corpus on the phone anyway, won't the index and model be relatively small in comparison?

sriram_malhar 5y ago
No, sorry for the confusion. I want to be able to type “kanun” or “qanoon”, and have it infer the Hindi word “कानून”, which is an indexed word. It is not necessary that there is a one to one correspondence between words. Sometimes two english words may represent one hindi word, or vice versa. I believe I can build up a decent sized training/validation set, for example from Bollywood song lyric databases written in English, and mapping them to the Hindi equivalent (or Tamil, Bengali, etc). As for your last question, I dont know, since I haver implemented an ML model in practice. I saw a tutorial on Bert this morning, where a word has 768 features. That itself sounds huge, leave alone the model itself.

talolard 5y ago

A non ml way to approach this is to use phonetic distance, e.g. qanoon and kanun sound the same so they are close. There is an algorithm called Soundex with python implementations you can try.

sriram_malhar 5y ago
Refer what I said about edit distance. Traditional methods don't work well at all. The fundamental difference between my problem and traditional spelling correction algos is that in the latter, there is a canonical correct spelling to be used as a reference. In my problem, there isn't. There are different approximate ways of spelling out most hindi words ... there is no one correct way. There are common patterns, sure, but it is too tedious to encode all the variations.