4 ms·

I'm not a fan of google but you can do exactly what this search engine does by curating your own list of domains to search against. https://programmablesearche

I'm not a fan of google but you can do exactly what this search engine does by curating your own list of domains to search against. https://programmablesearchengine.google.com https://programmablesearchengine.google.com

jfoster 4y ago
Interesting. This looks like Custom Search Engines evolved into this? I can't tell whether this is a neglected Google product that they were going to refresh but lost interest in, or something that is undergoing a breath of fresh air. As you say, I was able to add a list of domains and get some pretty decent results from it. The UI makes me feel like Google are not interested in making it a truly successful product, though.
zxwrt 4y ago
It requires google account, has tracking and isn't open-source. I'd say it is no go
- the_biot 4y ago
  Add to that the likelihood that Google will just randomly cancel the product one day. Why invest time in this Google product?
  - bastardoperator 4y ago
    It's been around forever, but your concern is real. Who's to say an OSS project won't get archived, or removed from the internet? Why invest time into anything when it will all be replaced eventually? edit: Looks like this OSS project was launched and cancelled in a single day.
    - the_biot 4y ago
      > Looks like this OSS project was launched and cancelled in a single day. Touché :-) However the idea of a federated search is some measure of protection against that. If it ever happens.
danskeren 4y ago
Some downsides with this approach: - search queries are performed directly from the clients computer so can't protect their privacy (since Custom Search JSON API have a daily limit of 10k queries) - forced to use javascript, and the way it's implemented makes it difficult if not impossible to do even basic things like the loading animation cards - ads are loaded from an iframe so you can't do any styling (except extremely limited options that they make available in their settings, but no matter what then it will be very ugly if you want to have a light/dark theme) But there are of course many benefits as well, such as it being 'free' (Bing is ridiculously expensive IMO, and feels impossible to join their ad network to offset the costs.. which might explain why you see countless Bing proxies shut down after a few months) and search results are no doubt better than the ones you'd get from Bing.
a5huynh 4y ago
For those looking for an alternative to that, I've been building a self-hosted search engine that crawls what you want based on a basic set of rules. It can be a list of domains, a very specific list of URLs, and/or even some basic regexes. https://github.com/a5huynh/spyglass https://github.com/a5huynh/spyglass
- imiric 4y ago
  Great project! Given a local archive of Wikipedia and other sources, this can be very powerful. Which raises the question: does archive.org offer their Wayback Machine index for download anywhere? Technically, why should anyone go through the trouble of crawling the web if archive.org has been doing it for years, and likely has one of the best indexes around? I've seen some 3rd-party downloaders for specific sites, but I'd like the full thing. Yes, I realize it's probably petabytes of data, but maybe it could be trimmed down to just the most recent crawls. If there was a way of having that index locally, it would make a very powerful search engine with a tool like yours.

jfoster 4y ago

Interesting. This looks like Custom Search Engines evolved into this? I can't tell whether this is a neglected Google product that they were going to refresh but lost interest in, or something that is undergoing a breath of fresh air. As you say, I was able to add a list of domains and get some pretty decent results from it. The UI makes me feel like Google are not interested in making it a truly successful product, though.

zxwrt 4y ago

It requires google account, has tracking and isn't open-source. I'd say it is no go

the_biot 4y ago
Add to that the likelihood that Google will just randomly cancel the product one day. Why invest time in this Google product?
- bastardoperator 4y ago
  It's been around forever, but your concern is real. Who's to say an OSS project won't get archived, or removed from the internet? Why invest time into anything when it will all be replaced eventually? edit: Looks like this OSS project was launched and cancelled in a single day.
  - the_biot 4y ago
    > Looks like this OSS project was launched and cancelled in a single day. Touché :-) However the idea of a federated search is some measure of protection against that. If it ever happens.

danskeren 4y ago

Some downsides with this approach: - search queries are performed directly from the clients computer so can't protect their privacy (since Custom Search JSON API have a daily limit of 10k queries) - forced to use javascript, and the way it's implemented makes it difficult if not impossible to do even basic things like the loading animation cards - ads are loaded from an iframe so you can't do any styling (except extremely limited options that they make available in their settings, but no matter what then it will be very ugly if you want to have a light/dark theme) But there are of course many benefits as well, such as it being 'free' (Bing is ridiculously expensive IMO, and feels impossible to join their ad network to offset the costs.. which might explain why you see countless Bing proxies shut down after a few months) and search results are no doubt better than the ones you'd get from Bing.

a5huynh 4y ago

For those looking for an alternative to that, I've been building a self-hosted search engine that crawls what you want based on a basic set of rules. It can be a list of domains, a very specific list of URLs, and/or even some basic regexes. https://github.com/a5huynh/spyglass https://github.com/a5huynh/spyglass

imiric 4y ago
Great project! Given a local archive of Wikipedia and other sources, this can be very powerful. Which raises the question: does archive.org offer their Wayback Machine index for download anywhere? Technically, why should anyone go through the trouble of crawling the web if archive.org has been doing it for years, and likely has one of the best indexes around? I've seen some 3rd-party downloaders for specific sites, but I'd like the full thing. Yes, I realize it's probably petabytes of data, but maybe it could be trimmed down to just the most recent crawls. If there was a way of having that index locally, it would make a very powerful search engine with a tool like yours.