3 ms·

Some neat stuff here. Only VERY briefly mentioned (so briefly I missed it at first) however: Substituting Yandex for Google is great for many use cases. Being R

Some neat stuff here. Only VERY briefly mentioned (so briefly I missed it at first) however: Substituting Yandex for Google is great for many use cases. Being Russian, Yandex is no doubt heavily censored, but only for things important to Russian politics. Ironically, this means that for non-Russian users, it's considerably LESS censored than Google, which has SEVERELY crippled its search in recent years in the name of politics, "politics," DMCA madness, "right to be forgotten" etc. The image search is especially impressive. Remember when Google Images used to give you actual results when trying to find the source of an obscure image? Yandex still does, and it does a bunch of other neat things too, like automatically trying to transcribe text from an image if it's text-heavy. My instinct is that a lot of this capacity exists in Google Images, but is either mostly hidden from the user or deliberately hobbled to stop the oh so evil content pirates. Zero privacy of course. Assume the Russian government is watching in realtime as you hammer in another inane search. But for some use cases that's fine.

userbinator 6y ago
Bing is also significantly less censored, especially for adult content, but also has a smaller index than Google and less operators.
- Logon90 6y ago
  Yeah, heard that is mostly why it still exists.
derefr 6y ago
Is there a search portal that does backend-side searches of all these politically-disjoint large search providers, and then merges and deduplicates the result?
- eli 6y ago
  Hah that was a big thing in the mid 90s. And, remarkably, this site still seems to be going strong https://www.dogpile.com/ https://www.dogpile.com/ I thought it was a really dumb name at the time, but I remembered it some 20 years later.
- 1vuio0pswjnm7 6y ago
  I have actually been working on a prototype for something similar to the "meta-search" engines of the 90s, but the intent is to deal with the present situation of search engines placing limits on number of results, and to rid SERPs of needless cruft (Javascript, CSS, advertising, etc.) making them look more standardised according to personal taste. I never really liked meta-search engines because they were too slow to return results. Instead this script lets me query search engines directly, one or more at a time, in succession, or all at the same time, and merge the results into simple, aesthetically-pleasing HTML files. I use a text-only browser to read HTML so what I am describing here is not designed with "modern" browsers in mind. The approach I take to avoid search result limits is that I search from the command line and store the result URLs in standardised "search results files" in a "search directory", one file per unique query. When I reach the results limit for the particular search engine, I can repeat the search on another search engine. The search results file is created for two reasons: 1. it allows me to strip out all the cruft from SERPs and mix results from different search engines into one file that looks great in a text-only browser, and 2. it tells me where I left off for each search engine, so I can continue any search at a later time, i.e., get more results. The script reads from the search directory and presents me with a menu of numbered searches. I continue a search at any time by selecting a number. For example: 1 this is an example 2 foo 3 foo bar 4 foo bar baz I typically browse results by pointing the text-only browser at the search directory. The search result files are each named according to the URL-encoded search string. The file format is very simple. There are three type of lines: 1. a title tag for the search query, 2. a link for each result URL, and 3. an HTML comment for each HTTP request, indicating to the script where I left off, i.e., the last result number; this is more or less equivalent to a "Next page" or "More results" link. Each result URL and comment is prefixed with a search engine identifier, i.e., a prefix. Thus the script can read the search results file comments and I can easily see which results came from XYZ search engine versus ABC search engine. <title>this is an example</title>  X <a href="https://example.com">https://example.com</a><br>   A <a href="https://example.net/index.html">https://example.net/index.html</a><br>  This example search results file above shows 1 result retrieved from XYZ search engine (prefix "X") and another result retrieved from ABC search engine (prefix "A"). It shows comments indicating where to continue the search for each search engine. Normally there would be around 50-100 result URLs per HTTP request. This approach assumes the search result limits are temporal, i.e., they are limits on how many results can be retrieved in a given period. That may not be true for every search engine.
1vuio0pswjnm7 6y ago
"Zero privacy of course." Surely this is not suggesting that Google searches afford any privacy. ;) While it might not be the Russian government who is watching, Google searches are certainly not ephemeral nor free from analysis in real-time. Aside from "things important to Russian politics", I would guess Google on behalf of its customers, who could be anyone, including governments, is far more interested in what someone is searching than the Russian government. The point I am getting at here is that there is privacy from a government and there is privacy from a company. Each is a form of privacy, but only the company is in the business of commercialising the information it derives from violating privacy. (Not to mention that, at least in the US, the government is subject to a body of privacy law that does not apply to companies.) Both the government and the company may violate privacy in the interests of staying in power. They could, e.g., suppress certain information when it is in their interests. However only the ad services company has the additional motivation to collect information to generate profits. The government may believe it has no choice but to monitor web search. The company OTOH freely chooses to monitor web search, as a business. Anyway, here is a question I have about Yandex. Google, Bing and other search engines such as DuckDuckGo now limit the number of results that can be retrieved in one session. Based on personal observation, with Google the ceiling is currently 300, with Bing and DDG, it's more like 250.^1 I wonder if Yandex is doing the same. These limits by Google, Bing, DDG, etc. are eliminating "discoverability" via searching the web. "General" searches that would yield more than 300 results will not return more than 300 results. Users collecting large numbers of results from a general search is effectively prohibited. Google's idea of discovery is "I'm feeling lucky". Of all the silly changes Google has made to search, the most useless feature persists. Perhaps "broad" searches do not benefit an online ad services business as much as more specific searches do. Limiting total results also puts more pressure on websites to try to be listed within the first 300. (Solution: Buy ads from Google.) A website who is at position 301 is undiscoverable thanks to Google's inexplicable truncation. Interestingly, on some, perhaps all, of their different "UI's", Google no longer numbers results. 1. To illustrate the truncation, try a search for a common string that would appear in the <title> tag of more than 300 pages on the web. https://www.google.com/search?q=title:[common https://www.google.com/search?q=title:[common string]&num=100&filter=0
- mgraczyk 6y ago
EE84M3i 6y ago
Mentioned in the article, but Yandex also does some degree of facial recognition (or at least similarity) when doing reverse image searches. It works surprisingly well.
Abishek_Muthian 6y ago
I haven't used Yandex much, But I started noting down failed search results according to the search engines in the hopes of one day creating a 'Search Engine Wall of Shame' to provide genuine feedback to the Search Engines in the areas they could improve[1]. I'll try Yandex for such queries too. [1] Those who are interested in a 'Search Engine Wall of Shame', URL for the discussion in my profile(#207).
RinTohsaka 6y ago
Is there something I could self host that would perform aa well as yandex?

userbinator 6y ago

Bing is also significantly less censored, especially for adult content, but also has a smaller index than Google and less operators.

Logon90 6y ago
Yeah, heard that is mostly why it still exists.

derefr 6y ago

Is there a search portal that does backend-side searches of all these politically-disjoint large search providers, and then merges and deduplicates the result?

eli 6y ago
Hah that was a big thing in the mid 90s. And, remarkably, this site still seems to be going strong https://www.dogpile.com/ https://www.dogpile.com/ I thought it was a really dumb name at the time, but I remembered it some 20 years later.
1vuio0pswjnm7 6y ago
I have actually been working on a prototype for something similar to the "meta-search" engines of the 90s, but the intent is to deal with the present situation of search engines placing limits on number of results, and to rid SERPs of needless cruft (Javascript, CSS, advertising, etc.) making them look more standardised according to personal taste. I never really liked meta-search engines because they were too slow to return results. Instead this script lets me query search engines directly, one or more at a time, in succession, or all at the same time, and merge the results into simple, aesthetically-pleasing HTML files. I use a text-only browser to read HTML so what I am describing here is not designed with "modern" browsers in mind. The approach I take to avoid search result limits is that I search from the command line and store the result URLs in standardised "search results files" in a "search directory", one file per unique query. When I reach the results limit for the particular search engine, I can repeat the search on another search engine. The search results file is created for two reasons: 1. it allows me to strip out all the cruft from SERPs and mix results from different search engines into one file that looks great in a text-only browser, and 2. it tells me where I left off for each search engine, so I can continue any search at a later time, i.e., get more results. The script reads from the search directory and presents me with a menu of numbered searches. I continue a search at any time by selecting a number. For example: 1 this is an example 2 foo 3 foo bar 4 foo bar baz I typically browse results by pointing the text-only browser at the search directory. The search result files are each named according to the URL-encoded search string. The file format is very simple. There are three type of lines: 1. a title tag for the search query, 2. a link for each result URL, and 3. an HTML comment for each HTTP request, indicating to the script where I left off, i.e., the last result number; this is more or less equivalent to a "Next page" or "More results" link. Each result URL and comment is prefixed with a search engine identifier, i.e., a prefix. Thus the script can read the search results file comments and I can easily see which results came from XYZ search engine versus ABC search engine. <title>this is an example</title>  X <a href="https://example.com">https://example.com</a><br>   A <a href="https://example.net/index.html">https://example.net/index.html</a><br>  This example search results file above shows 1 result retrieved from XYZ search engine (prefix "X") and another result retrieved from ABC search engine (prefix "A"). It shows comments indicating where to continue the search for each search engine. Normally there would be around 50-100 result URLs per HTTP request. This approach assumes the search result limits are temporal, i.e., they are limits on how many results can be retrieved in a given period. That may not be true for every search engine.

1vuio0pswjnm7 6y ago

"Zero privacy of course." Surely this is not suggesting that Google searches afford any privacy. ;) While it might not be the Russian government who is watching, Google searches are certainly not ephemeral nor free from analysis in real-time. Aside from "things important to Russian politics", I would guess Google on behalf of its customers, who could be anyone, including governments, is far more interested in what someone is searching than the Russian government. The point I am getting at here is that there is privacy from a government and there is privacy from a company. Each is a form of privacy, but only the company is in the business of commercialising the information it derives from violating privacy. (Not to mention that, at least in the US, the government is subject to a body of privacy law that does not apply to companies.) Both the government and the company may violate privacy in the interests of staying in power. They could, e.g., suppress certain information when it is in their interests. However only the ad services company has the additional motivation to collect information to generate profits. The government may believe it has no choice but to monitor web search. The company OTOH freely chooses to monitor web search, as a business. Anyway, here is a question I have about Yandex. Google, Bing and other search engines such as DuckDuckGo now limit the number of results that can be retrieved in one session. Based on personal observation, with Google the ceiling is currently 300, with Bing and DDG, it's more like 250.^1 I wonder if Yandex is doing the same. These limits by Google, Bing, DDG, etc. are eliminating "discoverability" via searching the web. "General" searches that would yield more than 300 results will not return more than 300 results. Users collecting large numbers of results from a general search is effectively prohibited. Google's idea of discovery is "I'm feeling lucky". Of all the silly changes Google has made to search, the most useless feature persists. Perhaps "broad" searches do not benefit an online ad services business as much as more specific searches do. Limiting total results also puts more pressure on websites to try to be listed within the first 300. (Solution: Buy ads from Google.) A website who is at position 301 is undiscoverable thanks to Google's inexplicable truncation. Interestingly, on some, perhaps all, of their different "UI's", Google no longer numbers results. 1. To illustrate the truncation, try a search for a common string that would appear in the <title> tag of more than 300 pages on the web. https://www.google.com/search?q=title:[common https://www.google.com/search?q=title:[common string]&num=100&filter=0

mgraczyk 6y ago

EE84M3i 6y ago

Mentioned in the article, but Yandex also does some degree of facial recognition (or at least similarity) when doing reverse image searches. It works surprisingly well.

Abishek_Muthian 6y ago

I haven't used Yandex much, But I started noting down failed search results according to the search engines in the hopes of one day creating a 'Search Engine Wall of Shame' to provide genuine feedback to the Search Engines in the areas they could improve[1]. I'll try Yandex for such queries too. [1] Those who are interested in a 'Search Engine Wall of Shame', URL for the discussion in my profile(#207).

RinTohsaka 6y ago

Is there something I could self host that would perform aa well as yandex?