6 ms·

I think you've done something very interesting that makes me rethink the idea of a "search engine". For all the data Google has, and all the AI and technology,

I think you've done something very interesting that makes me rethink the idea of a "search engine". For all the data Google has, and all the AI and technology, their search isn't very "smart" at allowing you to contextualize or filter the searches very well. For example, a button to remove social, to offer meta-data (such as whether a question is answered/unanswered) or offer features that might help me find what I am looking for. Google has a huge index, but naturally, there's going to be 'noise', and maybe I want to see what's on page 50, or want to try and 'filter' some of the more popular sites. Their UI for enabling this is non-existent. Another example: Google BigQuery has the entire Reddit archive, they could surface meta-data such as the number of comments or whether entire comment chains are deleted (although, doing that against Reddit's API at scale would be a very difficult challenge and perhaps not welcomed by reddit). But to me, as a user of search, that might influence what content I end up viewing, because perhaps a thread with 0 comments isn't so useful to me, or a thread full of [deleted] comments is worthless and should be de-indexed (and deciding what is valuable is a very hard question to answer).

anigbrowl 6y ago
I think all search engines fail in providing a list of ordered results. What you actually want is a look at the graph that they were plucked from. As things stand, it's not just a case of being unable to see the forest for the trees, but being unable to see the forest because you are forced to infer its structure from a small basket of leaves. The only company I've seen attempt anything like this is the (criminally under-recognized) carrotsearch.
- wizzwizz4 6y ago
  Do you mean https://search.carrot2.org/ https://search.carrot2.org/?
  - Red_Leaves_Flyy 6y ago
    If he does, I can see why it's not popular. I tried a single query and was told my search limits been exceeded.
jeffbee 6y ago
I think you are _radically_ underestimating the cost (and overestimating the benefit) of adding a feature to Google's index. When you are writing a small-scale search system over a specialized corpus where the index fits in the memory of a single machine it is easy to imagine and realize nifty features. When your search engine draws 100MW you have a completely different perspective.
- zippy5 6y ago
  You're telling me that google makes $30 for every financial service ad I click but are cheaping out on the signal processing? Is the decline search result quality in the last couple years the result of a business analyst arbitraging sensitivity to result quality vs compute costs?
  - jeffbee 6y ago
    Only randos on HN think search quality is declining. If the search quality metrics that they use internally suffered a sustained decline, everyone from Sundar down would be fired.
    - nwienert 6y ago
      They are declining, and I’ve met many non HN randos who agree. Quality doesn’t correlate with their metrics; likely some sort of mix of “engagement” and mostly how much they make from ads.
      - mD5pPxMcS6fVWKE 6y ago
        Confirm that. A lot of spam and SEO recently.
        emerongi 6y ago
        A whole ton of SEO. Any keyword from which you can make reasonable amounts of money will be cluttered with sites made by SEO people, who are really good at gaming the index, but really bad at anything else (i.e. actually giving proper information on the keyword you're searching for). Spam has become an issue again in my native language. For a couple of years Google did a great job at removing spam, but somehow in the past 2 years it has resurfaced.
polote 6y ago
> For all the data Google has, and all the AI and technology, their search isn't very "smart" at allowing you to contextualize or filter the searches very well. Google exists to make money from 99.9% of users, the ones who don't read HN, so that's pretty normal if you don't find it accurate
BbzzbB 6y ago
While I do not agree with storing user-deleted information, you can change the "reddit" URL to "removeddit" to see most of the comments. Alternatively, "ceddit" will only show mod-deleted comments.

anigbrowl 6y ago

I think all search engines fail in providing a list of ordered results. What you actually want is a look at the graph that they were plucked from. As things stand, it's not just a case of being unable to see the forest for the trees, but being unable to see the forest because you are forced to infer its structure from a small basket of leaves. The only company I've seen attempt anything like this is the (criminally under-recognized) carrotsearch.

wizzwizz4 6y ago
Do you mean https://search.carrot2.org/ https://search.carrot2.org/?
- Red_Leaves_Flyy 6y ago
  If he does, I can see why it's not popular. I tried a single query and was told my search limits been exceeded.

jeffbee 6y ago

I think you are _radically_ underestimating the cost (and overestimating the benefit) of adding a feature to Google's index. When you are writing a small-scale search system over a specialized corpus where the index fits in the memory of a single machine it is easy to imagine and realize nifty features. When your search engine draws 100MW you have a completely different perspective.

zippy5 6y ago
You're telling me that google makes $30 for every financial service ad I click but are cheaping out on the signal processing? Is the decline search result quality in the last couple years the result of a business analyst arbitraging sensitivity to result quality vs compute costs?
- jeffbee 6y ago
  Only randos on HN think search quality is declining. If the search quality metrics that they use internally suffered a sustained decline, everyone from Sundar down would be fired.
  - nwienert 6y ago
    They are declining, and I’ve met many non HN randos who agree. Quality doesn’t correlate with their metrics; likely some sort of mix of “engagement” and mostly how much they make from ads.
    - mD5pPxMcS6fVWKE 6y ago
      Confirm that. A lot of spam and SEO recently.
      - emerongi 6y ago
        A whole ton of SEO. Any keyword from which you can make reasonable amounts of money will be cluttered with sites made by SEO people, who are really good at gaming the index, but really bad at anything else (i.e. actually giving proper information on the keyword you're searching for). Spam has become an issue again in my native language. For a couple of years Google did a great job at removing spam, but somehow in the past 2 years it has resurfaced.

polote 6y ago

> For all the data Google has, and all the AI and technology, their search isn't very "smart" at allowing you to contextualize or filter the searches very well. Google exists to make money from 99.9% of users, the ones who don't read HN, so that's pretty normal if you don't find it accurate

BbzzbB 6y ago

While I do not agree with storing user-deleted information, you can change the "reddit" URL to "removeddit" to see most of the comments. Alternatively, "ceddit" will only show mod-deleted comments.