8 ms·

Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most

by xipho 5y ago

Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring... you get the idea. This was a little different, extremely few results, but a couple of them really made me grin, and all(?) made me curious or raise an eyebrow or reflect on who/what might have been the source of the link, or remember some obscure connection from grad-school. So, if anything a crawled list of results worthy of ponder, thanks for this!

duckmysick 5y ago
I'm intrigued by this experiment but I can't visualize it. What do you mean by boring results? Would combing through a library (the one with paper books) also produce boring results? What's your ideal results?
- xattt 5y ago
  There’s certain grey literature that’s not captured in university library federated searches nor easily found with mainstream search engines.
  - xipho 5y ago
    There are decades of academic research not digitized. The digitization window used to only hit around 1990, I haven't looked at it hard recently, but I suspect this still remains true for many important journals. This is grey only to those who do not know how to use a library.
- xipho 5y ago
  Perhaps a counter example, something that is interesting. Anecdotally. This, of all things, is the top result in my search: https://tft.brainiac.com/archive/0303/msg00037.html https://tft.brainiac.com/archive/0303/msg00037.html. Which is strange to me because I don't recognize tft.brainiac. I click, it's a list of biological relationships among Hymenoptera, including a reference to genus of the wasps I studied, presumably in a biological relationship (host/parasite) context. I cataloged every relationship known at one point, so my brain wants to know where this come from, is it something I caught. Then I go look for more context, and find it's part of a thread about D&D(?) and hymenoptera, and it's epic, and a chunk of my morning is lost figuring out why and how this came to be.
  - duckmysick 5y ago
    Yes, thanks. That helps. If I understand it correctly, you're interested in bits and pieces of new information that's indirectly related to your object of interest. Degree 2 and 3 in Six Degrees of Kevin Bacon, so to speak. You know degree 0 like the back of your hand and you've seen almost everything closely connected. Finding novel, interesting things is getting more difficult. Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology.
    - xipho 5y ago
      Exactly. > Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology. Tongue in cheek- new app time, to facilitate this. It should have the name "Degree4". Entries can only be made if degrees 2 and 3 are "defined". Scoffs at degrees 5 and 6, just because. Startup developing can probably unethically seed content by mining https://www.everything2.com/ https://www.everything2.com/. Should use concepts of "AI" and "persistent homology"... profit! But no, I don't outside a mental note. Closest I would come would be adding '!! <some note>' to my potwiki text notes (see my past comments) if its something I want to have come back with a grep, or think might be interesting to explore "when I retire". If it's a scientific fact in my field after researching it further it would go into this https://taxonworks.org https://taxonworks.org (or its precursor).
- xipho 5y ago
  In part, by boring results I mean I instantly recognize the top results, and I know exactly what will be in them, and I know which ones will actually contain potentially interesting new stuff, i.e. _I didn't have to search for these, I'd go their directly_. Then next results are all obscure, and I've already visited them, and/or I know they are historical and not something I have to revisit. With this engine with at least 1/2 the links (to be fair there were < 20) I didn't recognize the URL at all, and it was clear in the text or the URL that there was an interesting bit to check out (i.e. what Google should have also returned after they barfed out the things I don't need to know about), but had never succinctly done in my experience. I suppose the magic in this engine would have to be alerting the searcher that they found more of this type of link, as once I visited the 10 or so sites they would fall back into the "been there, done that" link category that Google appends somewhere after the ads and "big" sites, mixed in with a million search term spam sites, etc.
trutannus 5y ago
> well we know Google isn't for search anymore Do you suggest anything better? As far as I can tell, all the other search engines are either repackaged Bing (ie: DuckDuck), or are just as bad.
- giancarlostoro 5y ago
  Not sure, but I remember when Google could find literally anything. Then they started adding a bunch of exceptions and crapped out their quality. I wonder how insanely different results would be to get the older Google Engine from the 2000s search result wise. I now have to play games with Google to find things. I feel like I do less than I used to for some reason.
  - bbarnett 5y ago
    The other day, I was searching for something, and google's suggested, on-site answers took up 1/2 the first page. All wrong. The actual search results were another 1/4 page of completely identical results, followed by google ad placed search results. I thought to myself, they've finally done it. Real responses are no longer first page. A lot of the cause for google getting crappy, is "ok google", another "all platforms are the same" form of sickness. No, a desktop is not a phone. No, voice searching is not the same as phone, or desktop.
    - foobarian 5y ago
      I was just thinking that they finally became Lycos. It's what all the search engines except Google looked like back in the early 2000s - ad laden cesspools of irrelevant search results and other content. And it's why we all switched to Google at the time.
      - habibur 5y ago
        It's time to disrupt the market. As Google can't compete with a newcomer that penalize ads on page.
mountain_peak 5y ago
Likewise, I co-maintain the only "fan" site on one of my all-time favourite composers/performers, and gave the engine a shot with a unique string query. While my text-heavy WP-driven site didn't seem to make the cut, the results were highly relevant in that they were links to former band members and collaborators - a couple of which I didn't realize existed. That being said, there were a few sites (including my own) I expected to be returned, but no dice. Still, a fascinating experiment that many at HN have been clamouring for.
- xipho 5y ago
  Exactly this. A couple results returned reference to obscure now-defunct newsletters and clubs, people that I know were historically important for past researchers, but only because this was my research forcus for so long would I have known this.
- marginalia_nu 5y ago
  The search engine doesn't actually do full text search, so maybe your query was too... unique. But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all. Secondly, are you behind cloudflare or some other big-name CDN? Because, as I mentioned in another comment, I can't crawl their pages without getting captchad until they approve of my humble request to be classified as a good bot. There are some other hosting providers I flat out block on a subnet level because they host a large amount of link farms. This is currently Alibaba, Psychz, eSited, Cloud Yuqu and 1Blu.
  - withinboredom 5y ago
    It’d be nice if you had a page to get the current index status for a domain.
    - marginalia_nu 5y ago
tentacleuno 5y ago
> well we know Google isn't for search anymore, If you're talking about the ads, I would bear in mind Google's whole business model is basically online advertising. Search is just the vehicle to deliver those ads; I'd say Google is pretty good at throwing things back.
- jevgeni 5y ago
  But what’s their UVP? I’d say quick and relevant search results. And that seems to be constantly degrading.
  - tentacleuno 5y ago
    Well, the unique value proposition is their gigantic index, really fast search and a bunch of other things. I'm not sure about the quality of the results, I just use DuckDuckGo these days, but IMO the unique technical advancements are pretty unique to Google.
    - r-w 5y ago
      > I just use DuckDuckGo these days Then w.r.t. quality of results, you're just using Bing: https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/#:~:text=most%20commonly%20from%20Bing https://help.duckduckgo.com/duckduckgo-help-pages/results/so...

by xipho 5y ago

duckmysick 5y ago

I'm intrigued by this experiment but I can't visualize it. What do you mean by boring results? Would combing through a library (the one with paper books) also produce boring results? What's your ideal results?

xattt 5y ago
There’s certain grey literature that’s not captured in university library federated searches nor easily found with mainstream search engines.
- xipho 5y ago
  There are decades of academic research not digitized. The digitization window used to only hit around 1990, I haven't looked at it hard recently, but I suspect this still remains true for many important journals. This is grey only to those who do not know how to use a library.
xipho 5y ago
Perhaps a counter example, something that is interesting. Anecdotally. This, of all things, is the top result in my search: https://tft.brainiac.com/archive/0303/msg00037.html https://tft.brainiac.com/archive/0303/msg00037.html. Which is strange to me because I don't recognize tft.brainiac. I click, it's a list of biological relationships among Hymenoptera, including a reference to genus of the wasps I studied, presumably in a biological relationship (host/parasite) context. I cataloged every relationship known at one point, so my brain wants to know where this come from, is it something I caught. Then I go look for more context, and find it's part of a thread about D&D(?) and hymenoptera, and it's epic, and a chunk of my morning is lost figuring out why and how this came to be.
- duckmysick 5y ago
  Yes, thanks. That helps. If I understand it correctly, you're interested in bits and pieces of new information that's indirectly related to your object of interest. Degree 2 and 3 in Six Degrees of Kevin Bacon, so to speak. You know degree 0 like the back of your hand and you've seen almost everything closely connected. Finding novel, interesting things is getting more difficult. Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology.
  - xipho 5y ago
    Exactly. > Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology. Tongue in cheek- new app time, to facilitate this. It should have the name "Degree4". Entries can only be made if degrees 2 and 3 are "defined". Scoffs at degrees 5 and 6, just because. Startup developing can probably unethically seed content by mining https://www.everything2.com/ https://www.everything2.com/. Should use concepts of "AI" and "persistent homology"... profit! But no, I don't outside a mental note. Closest I would come would be adding '!! <some note>' to my potwiki text notes (see my past comments) if its something I want to have come back with a grep, or think might be interesting to explore "when I retire". If it's a scientific fact in my field after researching it further it would go into this https://taxonworks.org https://taxonworks.org (or its precursor).
xipho 5y ago
In part, by boring results I mean I instantly recognize the top results, and I know exactly what will be in them, and I know which ones will actually contain potentially interesting new stuff, i.e. _I didn't have to search for these, I'd go their directly_. Then next results are all obscure, and I've already visited them, and/or I know they are historical and not something I have to revisit. With this engine with at least 1/2 the links (to be fair there were < 20) I didn't recognize the URL at all, and it was clear in the text or the URL that there was an interesting bit to check out (i.e. what Google should have also returned after they barfed out the things I don't need to know about), but had never succinctly done in my experience. I suppose the magic in this engine would have to be alerting the searcher that they found more of this type of link, as once I visited the 10 or so sites they would fall back into the "been there, done that" link category that Google appends somewhere after the ads and "big" sites, mixed in with a million search term spam sites, etc.

trutannus 5y ago

> well we know Google isn't for search anymore Do you suggest anything better? As far as I can tell, all the other search engines are either repackaged Bing (ie: DuckDuck), or are just as bad.

giancarlostoro 5y ago
Not sure, but I remember when Google could find literally anything. Then they started adding a bunch of exceptions and crapped out their quality. I wonder how insanely different results would be to get the older Google Engine from the 2000s search result wise. I now have to play games with Google to find things. I feel like I do less than I used to for some reason.
- bbarnett 5y ago
  The other day, I was searching for something, and google's suggested, on-site answers took up 1/2 the first page. All wrong. The actual search results were another 1/4 page of completely identical results, followed by google ad placed search results. I thought to myself, they've finally done it. Real responses are no longer first page. A lot of the cause for google getting crappy, is "ok google", another "all platforms are the same" form of sickness. No, a desktop is not a phone. No, voice searching is not the same as phone, or desktop.
  - foobarian 5y ago
    I was just thinking that they finally became Lycos. It's what all the search engines except Google looked like back in the early 2000s - ad laden cesspools of irrelevant search results and other content. And it's why we all switched to Google at the time.
    - habibur 5y ago
      It's time to disrupt the market. As Google can't compete with a newcomer that penalize ads on page.

mountain_peak 5y ago

Likewise, I co-maintain the only "fan" site on one of my all-time favourite composers/performers, and gave the engine a shot with a unique string query. While my text-heavy WP-driven site didn't seem to make the cut, the results were highly relevant in that they were links to former band members and collaborators - a couple of which I didn't realize existed. That being said, there were a few sites (including my own) I expected to be returned, but no dice. Still, a fascinating experiment that many at HN have been clamouring for.

xipho 5y ago
Exactly this. A couple results returned reference to obscure now-defunct newsletters and clubs, people that I know were historically important for past researchers, but only because this was my research forcus for so long would I have known this.
marginalia_nu 5y ago
The search engine doesn't actually do full text search, so maybe your query was too... unique. But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all. Secondly, are you behind cloudflare or some other big-name CDN? Because, as I mentioned in another comment, I can't crawl their pages without getting captchad until they approve of my humble request to be classified as a good bot. There are some other hosting providers I flat out block on a subnet level because they host a large amount of link farms. This is currently Alibaba, Psychz, eSited, Cloud Yuqu and 1Blu.
- withinboredom 5y ago
  It’d be nice if you had a page to get the current index status for a domain.
  - marginalia_nu 5y ago

tentacleuno 5y ago

> well we know Google isn't for search anymore, If you're talking about the ads, I would bear in mind Google's whole business model is basically online advertising. Search is just the vehicle to deliver those ads; I'd say Google is pretty good at throwing things back.

jevgeni 5y ago
But what’s their UVP? I’d say quick and relevant search results. And that seems to be constantly degrading.
- tentacleuno 5y ago
  Well, the unique value proposition is their gigantic index, really fast search and a bunch of other things. I'm not sure about the quality of the results, I just use DuckDuckGo these days, but IMO the unique technical advancements are pretty unique to Google.
  - r-w 5y ago
    > I just use DuckDuckGo these days Then w.r.t. quality of results, you're just using Bing: https://help.duckduckgo.com/duckduckgo-help-pages/results/sources/#:~:text=most%20commonly%20from%20Bing https://help.duckduckgo.com/duckduckgo-help-pages/results/so...