5 ms·

So why aren't there more search engines these days? Google is great but we constantly talk here about how it's losing its edge for certain kinds of more specifi

by tgb 8y ago

So why aren't there more search engines these days? Google is great but we constantly talk here about how it's losing its edge for certain kinds of more specific searches like technical ones. So seems like there's room for engines that are more tuned for special use cases and the ability to index web pages has only gotten cheaper since Google started. Has the size of the web made this impractical? Or do I just not know about these options?

hartator 8y ago
I think the transformation has been more subtle, App Stores SEO is a thing, and voice searches are becoming more common. Document searches is still Google though.
deleted 8y ago
[deleted]
larkeith 8y ago
Utter layman's guess: While indexing has gotten easier, the web has expanded exponentially in the meantime - likely far outstripping any technical gains. I would be surprised if indexing the modern web is feasible without significant time and capital. Also, I can think of a couple specialized engines that do exist: Google Scholar and Shodan. There are probably more I'm unfamiliar with.
- fragsworth 8y ago
  Not only is it infeasible, but there's so much garbage out there that we have no simple way to filter it out from scratch to the point that anyone could actually use it. Google and Bing have feedback loops with their users that prevent crap from rising to the top of search results. A plain index (or even using something like pagerank) wouldn't have this huge benefit, and you'd never be able to get your search engine off the ground.
auganov 8y ago
I'd love a search engine with more querying power limited to a niche. But I doubt limiting topical scope would do much to keep computational complexity from blowing up. A while back some startup was charging per search to work around that.
- greglindahl 8y ago
  blekko provided 2,000 niches and it wasn't enough to keep us alive. Also, you need a web-wide crawl to build a good niche engine.
- tgb 8y ago
  I wasn't really thinking a niche topic but a niche use-case. Google is bad at doing literal, verbatim string searches these days and all I want sometimes is like Google 2005ish era search, just PageRank based and no modern machine learning, etc. You can have it simpler than Google did at the time since SEO against your niche engine is unlikely, so tech wise you might be more like 1998 Google. The main bottleneck seems to be what others here are suggesting: there's just too much web to index these days unless you're Bing-scale or bigger.
  - auganov 8y ago
    Well that's sort of what I'm thinking of too. I'm talking about the idea that this might be doable by limiting topical scope. So for example you'd only index some portion of tech related sites.
  - wutbrodo 8y ago
    > Google is bad at doing literal, verbatim string searches these days and all I want sometimes is like Google 2005ish era search, just PageRank based and no modern machine learning, etc You can get most of the way to this with the verbatim option, but I think they make it difficult to make the default. > By using the Verbatim tool Google will not make the following changes: • Personalizing your search using websites you have visited before; • Including synonyms of your search terms; • Automatic spelling corrections; • Searching for words with the same stem e.g. “Shopping” when searched for “shop”; • Finding results that match similar terms to those in your query.
Bodet 8y ago
for me the image search by google is broken and I can't find useful stuff with it
polm23 8y ago
I think these exist, they've just become more specialized. Think of these sites you might have seen before: - alternativeto.net and similar - Google Scholar / Semantic Scholar - Every sandboxed social network (Facebook, Twitter, Tumblr) - A variety of Instagram searching sites - Alternative App stores for Android And there's room for plenty more sites like this. Attempting to return a good response for anything in the search box is a bottomless problem of unclear utility. Being more focused makes the work easier and the value for the user clearer.
jccalhoun 8y ago
I think it is hard to get as good as google without spending tons of money and/or time and then get the market share to start possibly make any money.
SerLava 8y ago
The problem is that Google and Bing really are duopolistic. You'd think that a monopoly could just break instantly if all it required was typing in a different URL. But modern search engines are reliant on machine learning on mind-bogglingly enormous troves of real human interaction data. If you truly outsmart Google by inventing a better mousetrap, it's worth fuck-all. Your solution will probably require more usage data than you will ever be able to collect, because nobody will use your search engine while it still produces poor results.
- greglindahl 8y ago
  This is the conventional wisdom. On the plus side, it means that those of us who want to build innovative search engines don't have much competition!
  - SerLava 8y ago
    Well, ML on 100 billion searches is probably going to more closely approximate user intent than a living brain in a tank, because that brain doesn't know what the hell "lkw attachment" means. Looks like german-english bilingual logistics professionals are looking for truck parts vendors, while teenage Americans are looking for hidden locations of laser focusing and enhancement devices within the video game Wolfenstein The New Order. Do you envision a second route to that answer?
    - greglindahl 8y ago
      Well, the interface offered by blekko's Izik tablet search engine was that it would show 2 categories in the answer, one related to automotive and one related to games.
      - wutbrodo 8y ago
        Google has done this for a while, but at the level of recognized entities, not at the level of individual queries. Is that how the engine you're referring to worked too? Eg, would the "lkw attachment" example above have partitioned results?
prevedmedved 8y ago
Because Google and Bing are in the advertising business, not the search engine business. (There is no "search engine industry".) Their goal is ad clickthrough, not accurate search results.
- carlivar 8y ago
  Sort of. They need good search results so users return and have a probability of ad click again.
InclinedPlane 8y ago
Google jumped into the market when it was still possible to compete. And they came to the table with a product that was faster, better, cheaper, and more easy to monetize. Google-style data centers reduced costs, sharding and map-reduce plus streamlined design improved speed, pagerank improved quality, low cost fast searches meant that low cost advertisements could still bring in a lot of RoI. From that kernel they grew to dominance, becoming synonymous with the very term "search". Now we live in a different era, one where search is integrated in everything on every platform and where replacing the default search engine is a huge uphill battle. Let's say that someone creates a better search engine, how would people actually use it? I'll tell you how most people who bothered to use it would integrate into their routines. Firstly they would still use google for everything day to day. They would still use google maps for directions. They would still use gmail for mail. They would still use google search as the semantic front-end for their browsing. Only after they performed a search using google that produced unsatisfactory results would they then pull out the better search engine and make use of it for that one isolated search. And that's the problem, because that scenario is very hard for the better search engine maker to monetize while google would continue reaping the major monitization haul for the vast majority of search uses for that user. And going from zero to completely integrating into a user's experience in the same way that google does now is not a realistic prospect for most startups.

by tgb 8y ago

hartator 8y ago

I think the transformation has been more subtle, App Stores SEO is a thing, and voice searches are becoming more common. Document searches is still Google though.

deleted 8y ago

[deleted]

larkeith 8y ago

Utter layman's guess: While indexing has gotten easier, the web has expanded exponentially in the meantime - likely far outstripping any technical gains. I would be surprised if indexing the modern web is feasible without significant time and capital. Also, I can think of a couple specialized engines that do exist: Google Scholar and Shodan. There are probably more I'm unfamiliar with.

fragsworth 8y ago
Not only is it infeasible, but there's so much garbage out there that we have no simple way to filter it out from scratch to the point that anyone could actually use it. Google and Bing have feedback loops with their users that prevent crap from rising to the top of search results. A plain index (or even using something like pagerank) wouldn't have this huge benefit, and you'd never be able to get your search engine off the ground.

auganov 8y ago

I'd love a search engine with more querying power limited to a niche. But I doubt limiting topical scope would do much to keep computational complexity from blowing up. A while back some startup was charging per search to work around that.

greglindahl 8y ago
blekko provided 2,000 niches and it wasn't enough to keep us alive. Also, you need a web-wide crawl to build a good niche engine.
tgb 8y ago
I wasn't really thinking a niche topic but a niche use-case. Google is bad at doing literal, verbatim string searches these days and all I want sometimes is like Google 2005ish era search, just PageRank based and no modern machine learning, etc. You can have it simpler than Google did at the time since SEO against your niche engine is unlikely, so tech wise you might be more like 1998 Google. The main bottleneck seems to be what others here are suggesting: there's just too much web to index these days unless you're Bing-scale or bigger.
- auganov 8y ago
  Well that's sort of what I'm thinking of too. I'm talking about the idea that this might be doable by limiting topical scope. So for example you'd only index some portion of tech related sites.
- wutbrodo 8y ago
  > Google is bad at doing literal, verbatim string searches these days and all I want sometimes is like Google 2005ish era search, just PageRank based and no modern machine learning, etc You can get most of the way to this with the verbatim option, but I think they make it difficult to make the default. > By using the Verbatim tool Google will not make the following changes: • Personalizing your search using websites you have visited before; • Including synonyms of your search terms; • Automatic spelling corrections; • Searching for words with the same stem e.g. “Shopping” when searched for “shop”; • Finding results that match similar terms to those in your query.

Bodet 8y ago

for me the image search by google is broken and I can't find useful stuff with it

polm23 8y ago

I think these exist, they've just become more specialized. Think of these sites you might have seen before: - alternativeto.net and similar - Google Scholar / Semantic Scholar - Every sandboxed social network (Facebook, Twitter, Tumblr) - A variety of Instagram searching sites - Alternative App stores for Android And there's room for plenty more sites like this. Attempting to return a good response for anything in the search box is a bottomless problem of unclear utility. Being more focused makes the work easier and the value for the user clearer.

jccalhoun 8y ago

I think it is hard to get as good as google without spending tons of money and/or time and then get the market share to start possibly make any money.

SerLava 8y ago

The problem is that Google and Bing really are duopolistic. You'd think that a monopoly could just break instantly if all it required was typing in a different URL. But modern search engines are reliant on machine learning on mind-bogglingly enormous troves of real human interaction data. If you truly outsmart Google by inventing a better mousetrap, it's worth fuck-all. Your solution will probably require more usage data than you will ever be able to collect, because nobody will use your search engine while it still produces poor results.

greglindahl 8y ago
This is the conventional wisdom. On the plus side, it means that those of us who want to build innovative search engines don't have much competition!
- SerLava 8y ago
  Well, ML on 100 billion searches is probably going to more closely approximate user intent than a living brain in a tank, because that brain doesn't know what the hell "lkw attachment" means. Looks like german-english bilingual logistics professionals are looking for truck parts vendors, while teenage Americans are looking for hidden locations of laser focusing and enhancement devices within the video game Wolfenstein The New Order. Do you envision a second route to that answer?
  - greglindahl 8y ago
    Well, the interface offered by blekko's Izik tablet search engine was that it would show 2 categories in the answer, one related to automotive and one related to games.
    - wutbrodo 8y ago
      Google has done this for a while, but at the level of recognized entities, not at the level of individual queries. Is that how the engine you're referring to worked too? Eg, would the "lkw attachment" example above have partitioned results?

prevedmedved 8y ago

Because Google and Bing are in the advertising business, not the search engine business. (There is no "search engine industry".) Their goal is ad clickthrough, not accurate search results.

carlivar 8y ago
Sort of. They need good search results so users return and have a probability of ad click again.

InclinedPlane 8y ago

Google jumped into the market when it was still possible to compete. And they came to the table with a product that was faster, better, cheaper, and more easy to monetize. Google-style data centers reduced costs, sharding and map-reduce plus streamlined design improved speed, pagerank improved quality, low cost fast searches meant that low cost advertisements could still bring in a lot of RoI. From that kernel they grew to dominance, becoming synonymous with the very term "search". Now we live in a different era, one where search is integrated in everything on every platform and where replacing the default search engine is a huge uphill battle. Let's say that someone creates a better search engine, how would people actually use it? I'll tell you how most people who bothered to use it would integrate into their routines. Firstly they would still use google for everything day to day. They would still use google maps for directions. They would still use gmail for mail. They would still use google search as the semantic front-end for their browsing. Only after they performed a search using google that produced unsatisfactory results would they then pull out the better search engine and make use of it for that one isolated search. And that's the problem, because that scenario is very hard for the better search engine maker to monetize while google would continue reaping the major monitization haul for the vast majority of search uses for that user. And going from zero to completely integrating into a user's experience in the same way that google does now is not a realistic prospect for most startups.