3 ms·

Is there a search portal that does backend-side searches of all these politically-disjoint large search providers, and then merges and deduplicates the result?

• eli 6y ago
  Hah that was a big thing in the mid 90s. And, remarkably, this site still seems to be going strong https://www.dogpile.com/ https://www.dogpile.com/ I thought it was a really dumb name at the time, but I remembered it some 20 years later.
• 1vuio0pswjnm7 6y ago
  I have actually been working on a prototype for something similar to the "meta-search" engines of the 90s, but the intent is to deal with the present situation of search engines placing limits on number of results, and to rid SERPs of needless cruft (Javascript, CSS, advertising, etc.) making them look more standardised according to personal taste. I never really liked meta-search engines because they were too slow to return results. Instead this script lets me query search engines directly, one or more at a time, in succession, or all at the same time, and merge the results into simple, aesthetically-pleasing HTML files. I use a text-only browser to read HTML so what I am describing here is not designed with "modern" browsers in mind. The approach I take to avoid search result limits is that I search from the command line and store the result URLs in standardised "search results files" in a "search directory", one file per unique query. When I reach the results limit for the particular search engine, I can repeat the search on another search engine. The search results file is created for two reasons: 1. it allows me to strip out all the cruft from SERPs and mix results from different search engines into one file that looks great in a text-only browser, and 2. it tells me where I left off for each search engine, so I can continue any search at a later time, i.e., get more results. The script reads from the search directory and presents me with a menu of numbered searches. I continue a search at any time by selecting a number. For example: 1 this is an example 2 foo 3 foo bar 4 foo bar baz I typically browse results by pointing the text-only browser at the search directory. The search result files are each named according to the URL-encoded search string. The file format is very simple. There are three type of lines: 1. a title tag for the search query, 2. a link for each result URL, and 3. an HTML comment for each HTTP request, indicating to the script where I left off, i.e., the last result number; this is more or less equivalent to a "Next page" or "More results" link. Each result URL and comment is prefixed with a search engine identifier, i.e., a prefix. Thus the script can read the search results file comments and I can easily see which results came from XYZ search engine versus ABC search engine. <title>this is an example</title> <!-- X q=this%20is%20an%20example --> X <a href="https://example.com">https://example.com</a><br> <!-- X q=this%20is%20and%20example&p=2 --> <!-- A q=this%20is%20an%20example --> A <a href="https://example.net/index.html">https://example.net/index.html</a><br> <!-- A q=this%20is%20an%20example&start=2 --> This example search results file above shows 1 result retrieved from XYZ search engine (prefix "X") and another result retrieved from ABC search engine (prefix "A"). It shows comments indicating where to continue the search for each search engine. Normally there would be around 50-100 result URLs per HTTP request. This approach assumes the search result limits are temporal, i.e., they are limits on how many results can be retrieved in a given period. That may not be true for every search engine.