3 ms·
Probably easiest to talk about the NER for search keywords. I have found 1000-2000 examples per label is enough to get >80% accuracy. I don't remember the exact
by rpedela 8y ago
Probably easiest to talk about the NER for search keywords. I have found 1000-2000 examples per label is enough to get >80% accuracy. I don't remember the exact before/after adding pre-trained word vectors, but it was a 2-3% improvement and the current F1 score is ~86% I believe.
I have found it best to apply this to search tail queries (uncommon queries) because the relevance already sucks so any improvement is helpful. Often they are misspellings or unique wordings of common queries. If you have a product search engine, and you can recognize the product category then you have a much better chance of giving the user a relevant result even if the words are misspelled. If you can improve the overall relevance of tail queries, then revenue for the product search engine should increase as well.
I should also add that Spacy turns all words into vectors under-the-hood. Providing pre-trained word vectors accelerates the training process so you don't need as many examples (hopefully).