4 ms·
If I wanted to create a crawler that only crawled a single site and use this as my site's search engine, what would be the best way to do that?
by dariusj18 5y ago
If I wanted to create a crawler that only crawled a single site and use this as my site's search engine, what would be the best way to do that?
- chabad360 5y agoI would generate a JSON feed for the site and upload that (I am curious if this can detect duplicate records. Edit: it can using the UPSURT call), that way you could even have full text search. A crawler is probably not the best tool for this if you use some form of site generation, as you can also use it to generate the information that goes into Typesense.
- dariusj18 5y agoI also want pdfs and other files indexed
- jabo 5y agoTypesense operates on JSON documents. So you'd have to first extract the data from PDFs / office docs into JSON and then index it. There are libraries that let you do the conversion: https://github.com/modesty/pdf2json https://github.com/modesty/pdf2json https://github.com/microsoft/Simplify-Docx https://github.com/microsoft/Simplify-Docx
- yes_but_no 5y agoYou can index your crawl data with StorkSearch[0] then use its js interface for search. [0]https://stork-search.net/ https://stork-search.net/