4 ms·
I'm interested in something similar. What I've tried is to set up https://yacy.net/ https://yacy.net/ to index only those websites I care about. I set it up on
by rolisz 5y ago
I'm interested in something similar. What I've tried is to set up https://yacy.net/ https://yacy.net/ to index only those websites I care about. I set it up on my home NAS and I configured it to crawl Python, sklearn, and some blogs and it took around 1 hour.
Searching through the results was quite fast, but I found the results a bit lacking. Maybe with more tuning I could have obtained better answers, but while searching for things in the Python standard library, I would get lots of noise from other places where that function is used.
- wcerfgba 5y agoYes I was originally considering crawling and indexing on my local machine as well but once I realised I could build this with Searx and leave all of that work to Bing/Google/DDG/... it removed a lot of complexity from the project, and also means I don't have to worry about optimising the quality of the results. That said, there is scope to do more work around plugging all the result sets together: Searx's approach is quite naive and doesn't work as well in this use case as it does in it's original/intended use case.
- 6510 5y agoYaCy does full text search and indexes all pages. edit: But it works best for just a single set of websites. I wouldn't know how to build multiple indexes.