3 ms·

Hey! I'm actually really interested to work on a personal project along this front. About a year back I organzied some files from research into a massive reposi

Hey! I'm actually really interested to work on a personal project along this front. About a year back I organzied some files from research into a massive repository and I'd like to build a search engine to make it easier to query. Can you send me some pointers from where you started? Thank you for your time!

marginalia_nu 5y ago
Assuming you can't or don't want to use something like Lucene or Solr (which are quite resource hungry), here's how to build a search engine. It's takes a diverse mix of computer science and low level programming to get working, but nothing is exorbitantly difficult. One side of the problem is to extract keywords from the documents, probably a good idea to start with something along the lines of OpenNLP and use that for TextRank. Well that's roughly what I'm doing now anyway, after having tried a few other approaches. It depends a bit on the nature of the documents I guess. Proper NLP is slow, but dumb TF-IDF goes a long way. (just google these things and you'll figure it out). If you want full text search, you'll probably want to look into constructing a suffix tree as well. Or just fake it using N-grams, but you absolutely need some rules limiting which N-grams are valid search terms, or their numbers grow astronomically. POS-tagging is useful here if you can afford it. As for the index: First you need a component that maps keywords to integers. This only needs to be one way, I use a custom memory mapped hashtable, but depending on how much data you've got, there are probably easier alternatives. Next you need to set up a reverse index that maps words-id to document-id. Again, this is a bunch of memory mapped binary data, I use a mix of hash tables and sorted arrays for binary searches. That's sort of it. Then you draw the rest of the owl.
kordlessagain 5y ago
I’ve build something that uses Solr for indexing and some old ideas I had about crawling with Grub back in the day. The solution uses either the browser or a Selinum box to image the website and OCR to extract the text and bounding boxes. It will save the image in Google cloud and provides full text search and other Solr facet functionality for ranking. It’s not designed to crawl recursively, but it allows users to save both URLs and images they want to search for later. Menus work great and with some more work in extracting images from pages with OpenCV, it should allow for searching labels coming in from machine learning models. Happy to share with who ever wants to take a look. The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0 https://gitHub.com/kordless/grub-2.0
trillic 5y ago
There's a few main components - HTML parser. - Crawler. - Index. - Constraint solver. - Query language. - Ranker. - Front end. You can use libraries for as many portions as you like to bootstrap. It's a great personal project.

marginalia_nu 5y ago

Assuming you can't or don't want to use something like Lucene or Solr (which are quite resource hungry), here's how to build a search engine. It's takes a diverse mix of computer science and low level programming to get working, but nothing is exorbitantly difficult. One side of the problem is to extract keywords from the documents, probably a good idea to start with something along the lines of OpenNLP and use that for TextRank. Well that's roughly what I'm doing now anyway, after having tried a few other approaches. It depends a bit on the nature of the documents I guess. Proper NLP is slow, but dumb TF-IDF goes a long way. (just google these things and you'll figure it out). If you want full text search, you'll probably want to look into constructing a suffix tree as well. Or just fake it using N-grams, but you absolutely need some rules limiting which N-grams are valid search terms, or their numbers grow astronomically. POS-tagging is useful here if you can afford it. As for the index: First you need a component that maps keywords to integers. This only needs to be one way, I use a custom memory mapped hashtable, but depending on how much data you've got, there are probably easier alternatives. Next you need to set up a reverse index that maps words-id to document-id. Again, this is a bunch of memory mapped binary data, I use a mix of hash tables and sorted arrays for binary searches. That's sort of it. Then you draw the rest of the owl.

kordlessagain 5y ago

I’ve build something that uses Solr for indexing and some old ideas I had about crawling with Grub back in the day. The solution uses either the browser or a Selinum box to image the website and OCR to extract the text and bounding boxes. It will save the image in Google cloud and provides full text search and other Solr facet functionality for ranking. It’s not designed to crawl recursively, but it allows users to save both URLs and images they want to search for later. Menus work great and with some more work in extracting images from pages with OpenCV, it should allow for searching labels coming in from machine learning models. Happy to share with who ever wants to take a look. The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0 https://gitHub.com/kordless/grub-2.0

trillic 5y ago

There's a few main components - HTML parser. - Crawler. - Index. - Constraint solver. - Query language. - Ranker. - Front end. You can use libraries for as many portions as you like to bootstrap. It's a great personal project.