5 ms·

Custom Search Engine Built on Searx

79 points by wcerfgba 5y ago

greyivy 5y ago
I've seen this blog on here a couple of times now. Does anyone know what it's built on (or if it's using a specific theme)?
- wcerfgba 5y ago
  I'm using GitHub Pages and Jekyll. I wrote the theme from almost-scratch, I think I started with a basic theme and then ripped out everything I didn't like, and then started building up again. Due to limitations on available Jekyll plugins on GH Pages, I wrote some particularly spicy Liquid templating to handle my automatic backlinks and site tree construction: https://github.com/wcerfgba/jpreston.xyz/blob/master/_includes/tree.html https://github.com/wcerfgba/jpreston.xyz/blob/master/_includ...
  - greyivy 5y ago
    Thanks so much for the insight!
santamex 5y ago
I am looking for a real custom search engine for months now. I tried google programmable search engine, bing custom search and was even building scripts for duckduckgo, but nothing really worked for me. For example, I have a list of 7500 company websites and I want to search them regularly for keywords, like product names, company names mentioned on them or if they have a some references to certain industries. This could be a solution.
- wcerfgba 5y ago
  Not exactly the same but Syften (https://syften.com/ https://syften.com/) can do this kind of 'listening' for social platforms like HN, Reddit, ... . Possibly you could reach out to Michal about extending their ingest with arbitrary/custom websites?
  - santamex 5y ago
    I will do so, thanks for the hint.
    - akfaew 5y ago
      I'm happy to work with you on this. My email is michal@syften.com.
- deleted 5y ago
  [deleted]
circularfoyers 5y ago
It would be fantastic if this sort of functionality was upstreamed. I think it would really appeal to the audience that uses/hosts Searx.
- wcerfgba 5y ago
  I did open an issue on GitHub to gauge interest in upstreaming it :) If you like it, leave a thumbs up :) https://github.com/searx/searx/issues/2952 https://github.com/searx/searx/issues/2952
rolisz 5y ago
I'm interested in something similar. What I've tried is to set up https://yacy.net/ https://yacy.net/ to index only those websites I care about. I set it up on my home NAS and I configured it to crawl Python, sklearn, and some blogs and it took around 1 hour. Searching through the results was quite fast, but I found the results a bit lacking. Maybe with more tuning I could have obtained better answers, but while searching for things in the Python standard library, I would get lots of noise from other places where that function is used.
- wcerfgba 5y ago
  Yes I was originally considering crawling and indexing on my local machine as well but once I realised I could build this with Searx and leave all of that work to Bing/Google/DDG/... it removed a lot of complexity from the project, and also means I don't have to worry about optimising the quality of the results. That said, there is scope to do more work around plugging all the result sets together: Searx's approach is quite naive and doesn't work as well in this use case as it does in it's original/intended use case.
  - 6510 5y ago
    YaCy does full text search and indexes all pages. edit: But it works best for just a single set of websites. I wouldn't know how to build multiple indexes.
yewenjie 5y ago
FWIW, there is a fork of searx which has seen a lot of activity recently. https://github.com/searxng/searxng https://github.com/searxng/searxng
hargup 5y ago
Wow, this is exactly the way we have been thinking about Neera. Checkout https://hargup.substack.com/p/so-convenient-that-it-changes-the https://hargup.substack.com/p/so-convenient-that-it-changes-...
deleted 5y ago
[deleted]
marcus_pousette 5y ago
I have made an attempt to this, but far from complete. I manually indexed the search engines of different websites in a common natural language interface: When writing "date parser MIT license stars > 1000" It automatically suggests GitHub as a website that can support this query: https://quantleaf.com/?q=&t=query&run=false https://quantleaf.com/?q=&t=query&run=false You can see more examples on the website.
- codetrotter 5y ago
  On mobile your site shows me that GitHub can support that query, but I’d like to be able to then tap on it and be taken to the GitHub search in question. At least on iOS with Safari, I could not find a way to be taken to the GitHub results by tapping on anything.
  - marcus_pousette 5y ago
    When the results appear, you have to press the small play button at the right side, then you will be navigated to GitHub to see the expected search results. I tried it now on iOS with Safari. I tried "date parser mit license". It could be more intuitive though

79 points by wcerfgba 5y ago

I've seen this blog on here a couple of times now. Does anyone know what it's built on (or if it's using a specific theme)?

wcerfgba 5y ago
I'm using GitHub Pages and Jekyll. I wrote the theme from almost-scratch, I think I started with a basic theme and then ripped out everything I didn't like, and then started building up again. Due to limitations on available Jekyll plugins on GH Pages, I wrote some particularly spicy Liquid templating to handle my automatic backlinks and site tree construction: https://github.com/wcerfgba/jpreston.xyz/blob/master/_includes/tree.html https://github.com/wcerfgba/jpreston.xyz/blob/master/_includ...
- greyivy 5y ago
  Thanks so much for the insight!

santamex 5y ago

I am looking for a real custom search engine for months now. I tried google programmable search engine, bing custom search and was even building scripts for duckduckgo, but nothing really worked for me. For example, I have a list of 7500 company websites and I want to search them regularly for keywords, like product names, company names mentioned on them or if they have a some references to certain industries. This could be a solution.

wcerfgba 5y ago
Not exactly the same but Syften (https://syften.com/ https://syften.com/) can do this kind of 'listening' for social platforms like HN, Reddit, ... . Possibly you could reach out to Michal about extending their ingest with arbitrary/custom websites?
- santamex 5y ago
  I will do so, thanks for the hint.
  - akfaew 5y ago
    I'm happy to work with you on this. My email is michal@syften.com.
deleted 5y ago
[deleted]

circularfoyers 5y ago

It would be fantastic if this sort of functionality was upstreamed. I think it would really appeal to the audience that uses/hosts Searx.

wcerfgba 5y ago
I did open an issue on GitHub to gauge interest in upstreaming it :) If you like it, leave a thumbs up :) https://github.com/searx/searx/issues/2952 https://github.com/searx/searx/issues/2952

rolisz 5y ago

I'm interested in something similar. What I've tried is to set up https://yacy.net/ https://yacy.net/ to index only those websites I care about. I set it up on my home NAS and I configured it to crawl Python, sklearn, and some blogs and it took around 1 hour. Searching through the results was quite fast, but I found the results a bit lacking. Maybe with more tuning I could have obtained better answers, but while searching for things in the Python standard library, I would get lots of noise from other places where that function is used.

wcerfgba 5y ago
Yes I was originally considering crawling and indexing on my local machine as well but once I realised I could build this with Searx and leave all of that work to Bing/Google/DDG/... it removed a lot of complexity from the project, and also means I don't have to worry about optimising the quality of the results. That said, there is scope to do more work around plugging all the result sets together: Searx's approach is quite naive and doesn't work as well in this use case as it does in it's original/intended use case.
- 6510 5y ago
  YaCy does full text search and indexes all pages. edit: But it works best for just a single set of websites. I wouldn't know how to build multiple indexes.

yewenjie 5y ago

FWIW, there is a fork of searx which has seen a lot of activity recently. https://github.com/searxng/searxng https://github.com/searxng/searxng

hargup 5y ago

Wow, this is exactly the way we have been thinking about Neera. Checkout https://hargup.substack.com/p/so-convenient-that-it-changes-the https://hargup.substack.com/p/so-convenient-that-it-changes-...

deleted 5y ago

[deleted]

marcus_pousette 5y ago

I have made an attempt to this, but far from complete. I manually indexed the search engines of different websites in a common natural language interface: When writing "date parser MIT license stars > 1000" It automatically suggests GitHub as a website that can support this query: https://quantleaf.com/?q=&t=query&run=false https://quantleaf.com/?q=&t=query&run=false You can see more examples on the website.

codetrotter 5y ago
On mobile your site shows me that GitHub can support that query, but I’d like to be able to then tap on it and be taken to the GitHub search in question. At least on iOS with Safari, I could not find a way to be taken to the GitHub results by tapping on anything.
- marcus_pousette 5y ago
  When the results appear, you have to press the small play button at the right side, then you will be navigated to GitHub to see the expected search results. I tried it now on iOS with Safari. I tried "date parser mit license". It could be more intuitive though