2 ms·
I'm doubtful that ElasticSearch, out of the box, is going to do much in the way of getting you want you want. It's fine for building an inverted text index and
by mindcrime 9y ago
I'm doubtful that ElasticSearch, out of the box, is going to do much in the way of getting you want you want. It's fine for building an inverted text index and letting you query it, but that assumes you have some ability to parse the data and submit it for indexing. If the data is truly "unknown" then your problem is how to get it into ES in the first place. And that would be true whether you used ES, Solr or just native Lucene.
Do you know if the data is at least in a text format? If it is, you could code up a job to walk through the data and extract usernames, phone numbers, etc. Something like OpenNLP would probably work well for you. If you want something a little fancier, like NER with automatic entity linking, you could use Apache Stanbol.
Keep in mind though, none of these tools are magic and none have human level AGI. You probably won't get 100% perfect matching for every single entity you care about. This is especially so using the pre-trained models. To get the best results, you'll need to human label some training data (a subset of your original data) and train your own model using the labeled data.
Once you're able to match all your various entities, then you might want to stick them in ES, Solr, etc. so you can do convenient queries - depending on what your use case is.
Note that if your data isn't in text to begin with (eg, if it's Word docs, PDF, etc.) you might be able to use Tika to extract the content as the first step in your processing pipeline.