4 ms·
Can you explain how you use FSTs and REs for Multi-word expression matching and Unknown word-guessing?
by bravura 8y ago
Can you explain how you use FSTs and REs for Multi-word expression matching and Unknown word-guessing?
- sergeio76 8y agoMWE: 1. given your corpus you run something like word2phrase algorithm on it. 2. Compute the phrases. 3. Compile them as one Min DFA and do left-most-longest match between the input sequence of tokens and the compiled DFA. We will include this as a separate API with phrases computed on some large corpus. Word-Guessing: 1. We compute a mapping as a finite state machine from a word to its properties. 2. we generalize the finite state machine by shrining some path and creating new finite state with the union of the properties so that a. no mistakes are made wrt the known words, b. the model is as general as possible. The result is a finite state machine that a. does not make mistakes wrt the known set of words and is b. as general as possible for typical unseen words. We will put description better than this comment once the models are published.