4 ms·

When will somebody replace Google (and Bing) with one that actually works again? Is the cost of entry that high? Is there no business model to permit one to op

by beezle 6y ago

When will somebody replace Google (and Bing) with one that actually works again? Is the cost of entry that high? Is there no business model to permit one to operate at some level of profit? I'm sure most here have been frustrated by the difficulty of getting "good" results on searches, even with modifiers. But what most troubles me is Google's memory/history has grown smaller and smaller, as if it has Alzheimers - searches that used to return results now bring back none.

Dumblydorr 6y ago
Too many sponsored links, and too much personalization. What happened to displaying the best information at the top in an easily scanned fashion? Maybe there would be more search engines in a less anti-competitive market... monopolies are hard to tackle.
toast0 6y ago
The cost of entry is pretty high. The web corpus is huge, which leads to follow on problems: it's expensive to fetch it, and to proccess it, and to host the resulting indexes. Fetching is tricky also because you're likely to get blocked from sites it you're too aggressive. To justify all that expense, you need a lot of users, but it will be hard to get those users because there is 20 years of 'search = google' to compete with. Yahoo search user testing from 10+ years ago found that users would prefer search results displayed with Google branding over search results with Yahoo branding, regardless of the search results. Maybe it wouldn't be so bad if it's Google vs a new hip name, but you have to somehow cultivate that hipness. Bing doesn't have it, Amazon tried doing websearch and quit pretty quick (but maybe it's used for Alexa?). You'd realistically need to build out an advertising platform too. Using Google's ad platform while trying to compete with their core market seems like a bad idea. Using Microsoft's ad platform is probably not going to a good experience, but maybe you can start with it.
- zuppy 6y ago
  there’s another big problem that I read about a few days ago: many sites are blocking all crawlers that are not google through robots.txt file.
jakub_g 6y ago
Cliqz tried to build index from scratch and be privacy-first, but they failed to reach user base big enough to survive the pandemic. https://cliqz.com/announcement.html https://cliqz.com/announcement.html
jldugger 6y ago
The common crawl is ~270 TiB uncompressed: https://commoncrawl.org/2020/12/nov-dec-2020-crawl-archive-now-available/ https://commoncrawl.org/2020/12/nov-dec-2020-crawl-archive-n... Which would be easy enough to store in 2020, but then you need to preprocess it in a way that is amenable to both search and result ranking. But lets say your indexing is super good and matches the compressed version: 80TiB. Throw that in EBS, and you're paying $6k/month just to store it. You also need CPU, and memory to actually compute from that data though! If we instead use i3 metal instances, you're looking at about $2700/month each, and you'll need 15 of them for 3x replication. $40k per month isn't bad, if you're a startup with VC funding. But... we also need network egress... All this just to be literally the common denominator search engine with zero users. So, how do your users get to you? In 2020, you have three major sources: browser searches, phone searches and direct traffic. If you want to be in the browsers, you're going to have to pay, and if you want to be default you must pay more than the incumbents who have their business model figured out. And bid at scales of roughly your Series B and C combined. Phone OSes, same deal: you need to be prepared to bid high, and in volume. Direct traffic is basically word of mouth / marketing driven, and for our common crawl search, we can assume is relatively nil. So even search traffic has an acquisition cost, and almost all of the sources run their own search engine that you would need to bid against. So this point you need to start thinking about revenue, because every query you get literally costs you money. We know that search engine ads work okay, since the user is clearly expressing intent. But those different users have very different values -- someone searching from an iPhone or MacBook Pro is likely more valuable to advertisers than a 10 year old Linux laptop running Firefox with Adblock and a Pi-Hole DNS server. And without traffic nobody's going to bother running campaigns on your platform. Alternative revenue strategies seem unlikely to work -- Google is free and Bing literally pays users, so subscription seems unworkable. You could try to find a niche, the way DDG has, and perhaps chisel away at market share slowly, but you'd need some content indexed that is unavailable to competitors, and that will come at a price.
ColinHayhurst 6y ago
Replacing isn't necessary, but we need alternatives. And that means independent indexing. You can alawys use more than one. There are only 9 companies (crawling & indexing) independently: Google, Bing, Yandex, Baidu, Sogou, Mojeek, Gigablast, Naver.kr, seznam.cz as listed here, with other search partners (of Google and Microsoft) https://twitter.com/SearchEngineMap/lists https://twitter.com/SearchEngineMap/lists Disclosure: I maintain that resource and work at Mojeek

by beezle 6y ago

Dumblydorr 6y ago

Too many sponsored links, and too much personalization. What happened to displaying the best information at the top in an easily scanned fashion? Maybe there would be more search engines in a less anti-competitive market... monopolies are hard to tackle.

toast0 6y ago

The cost of entry is pretty high. The web corpus is huge, which leads to follow on problems: it's expensive to fetch it, and to proccess it, and to host the resulting indexes. Fetching is tricky also because you're likely to get blocked from sites it you're too aggressive. To justify all that expense, you need a lot of users, but it will be hard to get those users because there is 20 years of 'search = google' to compete with. Yahoo search user testing from 10+ years ago found that users would prefer search results displayed with Google branding over search results with Yahoo branding, regardless of the search results. Maybe it wouldn't be so bad if it's Google vs a new hip name, but you have to somehow cultivate that hipness. Bing doesn't have it, Amazon tried doing websearch and quit pretty quick (but maybe it's used for Alexa?). You'd realistically need to build out an advertising platform too. Using Google's ad platform while trying to compete with their core market seems like a bad idea. Using Microsoft's ad platform is probably not going to a good experience, but maybe you can start with it.

zuppy 6y ago
there’s another big problem that I read about a few days ago: many sites are blocking all crawlers that are not google through robots.txt file.

jakub_g 6y ago

Cliqz tried to build index from scratch and be privacy-first, but they failed to reach user base big enough to survive the pandemic. https://cliqz.com/announcement.html https://cliqz.com/announcement.html

jldugger 6y ago

The common crawl is ~270 TiB uncompressed: https://commoncrawl.org/2020/12/nov-dec-2020-crawl-archive-now-available/ https://commoncrawl.org/2020/12/nov-dec-2020-crawl-archive-n... Which would be easy enough to store in 2020, but then you need to preprocess it in a way that is amenable to both search and result ranking. But lets say your indexing is super good and matches the compressed version: 80TiB. Throw that in EBS, and you're paying $6k/month just to store it. You also need CPU, and memory to actually compute from that data though! If we instead use i3 metal instances, you're looking at about $2700/month each, and you'll need 15 of them for 3x replication. $40k per month isn't bad, if you're a startup with VC funding. But... we also need network egress... All this just to be literally the common denominator search engine with zero users. So, how do your users get to you? In 2020, you have three major sources: browser searches, phone searches and direct traffic. If you want to be in the browsers, you're going to have to pay, and if you want to be default you must pay more than the incumbents who have their business model figured out. And bid at scales of roughly your Series B and C combined. Phone OSes, same deal: you need to be prepared to bid high, and in volume. Direct traffic is basically word of mouth / marketing driven, and for our common crawl search, we can assume is relatively nil. So even search traffic has an acquisition cost, and almost all of the sources run their own search engine that you would need to bid against. So this point you need to start thinking about revenue, because every query you get literally costs you money. We know that search engine ads work okay, since the user is clearly expressing intent. But those different users have very different values -- someone searching from an iPhone or MacBook Pro is likely more valuable to advertisers than a 10 year old Linux laptop running Firefox with Adblock and a Pi-Hole DNS server. And without traffic nobody's going to bother running campaigns on your platform. Alternative revenue strategies seem unlikely to work -- Google is free and Bing literally pays users, so subscription seems unworkable. You could try to find a niche, the way DDG has, and perhaps chisel away at market share slowly, but you'd need some content indexed that is unavailable to competitors, and that will come at a price.

ColinHayhurst 6y ago

Replacing isn't necessary, but we need alternatives. And that means independent indexing. You can alawys use more than one. There are only 9 companies (crawling & indexing) independently: Google, Bing, Yandex, Baidu, Sogou, Mojeek, Gigablast, Naver.kr, seznam.cz as listed here, with other search partners (of Google and Microsoft) https://twitter.com/SearchEngineMap/lists https://twitter.com/SearchEngineMap/lists Disclosure: I maintain that resource and work at Mojeek