3 ms·

Having worked on Google's indexing system, I can't imagine how much it would cost to write all of the crawling and indexing code from scratch, and then run it o

by KMag 5y ago

Having worked on Google's indexing system, I can't imagine how much it would cost to write all of the crawling and indexing code from scratch, and then run it on over the visible web. You need to bootstrap somehow form an existing index if you want to get anywhere in any reasonable time. If there's a market for alternative search engines out there, AWS is already serving so much of the web that Amazon should really provide crawling and basic indexing (for a fee, of course) so it's done once for everyone, with Amazon Lambda processing to allow search engines to customize their indexing. I'm not sure if it would make them much money, but it makes more sense to start up a search engine on AWS using Amazon's crawl and basic indexing vs. using a search engine competitor's crawl and indexing.

post-it 5y ago
That's an insanely good idea. Since behind the scenes, Amazon would know who owns each website that it's crawling, an AWS search database would inherently be better-curated and more trustworthy than anything an external crawler could put together. Having your website on AWS would be crucial for SEO. Awful for competition and whatnot.
- KMag 5y ago
  Google and Bing can already crawl AWS sites just fine. As long as Amazon doesn't provide any first-party search engine, just providing search engine infrastructure, I think it would be a net win for competition. As it stands, if you want to bootstrap your own search engine, you need to base your search engine on either Google or Bing's index, or perform the herculean task of making your own crawler+indexer+search engine. If Amazon can commoditize the back end, and let AWS-hosted search engines provide differentiators late in the indexing pipeline and on the search/serving side, I think we'd see more niche search engines spring up.
perardi 5y ago
You know what? Not that crazy of an idea. Amazon is a weirdly plausible competitor to Google. They have a surprisingly robust ad business, and obviously AWS. https://www.cnbc.com/2022/02/03/amazon-has-a-31-billion-a-year-advertising-business.html https://www.cnbc.com/2022/02/03/amazon-has-a-31-billion-a-ye...
guyzero 5y ago
Amazon ran A9.com as a search engine from 2004 to 2008. https://en.wikipedia.org/wiki/A9.com#A9.com_search_portal https://en.wikipedia.org/wiki/A9.com#A9.com_search_portal
- KMag 5y ago
  Yes, A9 was one of the competitors for some of my time at Google, but I'm talking about Amazon basically creating some search engine infrastructure and a framework for building search engines on top of. Selling shovels to the gold miners, in a sense. The crawling and initial indexing stages need a lot of scaling and performance tuning work, which is a core competency for the AWS engineers. Furthermore, essentially zero people say "I want to build a startup around my new crawl scheduling idea, but it's too time consuming and expensive to write the rest of the search engine." Many more people say "I have a new idea for ranking" or "I have a new idea for content extraction, but it's too expensive and time-consuming to write the rest of a search engine". Google and Bing already expose "custom search engines" which are basically just filters on top of their search engines. I'm talking about something similar, but much deeper customization. Also, I was at Goldman when Google Search Appliance was EoL'd. GS had plenty of internal content that needed to be indexed, absolutely needed to be kept private, and wasn't well indexed by any of the replacements GS evaluated. For companies that keep lots of documents in AWS storage, it would be good to open up the space for Google Search Appliance replacements. Amazon could write their own, but given that Google abandoned the space, the profit margins aren't great for a huge company. However, creating a search ecosystem within the AWS ecosystem makes the AWS ecosystem more attractive. Maybe forgoing the search profits and inviting in search providers would be a net win in selling more AWS compute and storage.
  - guyzero 5y ago
    Unfortunately I suspect this would be selling gold shovels to dirt miners and the economics of it are not compelling.
deleted 5y ago
[deleted]

by KMag 5y ago

post-it 5y ago

That's an insanely good idea. Since behind the scenes, Amazon would know who owns each website that it's crawling, an AWS search database would inherently be better-curated and more trustworthy than anything an external crawler could put together. Having your website on AWS would be crucial for SEO. Awful for competition and whatnot.

KMag 5y ago
Google and Bing can already crawl AWS sites just fine. As long as Amazon doesn't provide any first-party search engine, just providing search engine infrastructure, I think it would be a net win for competition. As it stands, if you want to bootstrap your own search engine, you need to base your search engine on either Google or Bing's index, or perform the herculean task of making your own crawler+indexer+search engine. If Amazon can commoditize the back end, and let AWS-hosted search engines provide differentiators late in the indexing pipeline and on the search/serving side, I think we'd see more niche search engines spring up.

perardi 5y ago

You know what? Not that crazy of an idea. Amazon is a weirdly plausible competitor to Google. They have a surprisingly robust ad business, and obviously AWS. https://www.cnbc.com/2022/02/03/amazon-has-a-31-billion-a-year-advertising-business.html https://www.cnbc.com/2022/02/03/amazon-has-a-31-billion-a-ye...

guyzero 5y ago

Amazon ran A9.com as a search engine from 2004 to 2008. https://en.wikipedia.org/wiki/A9.com#A9.com_search_portal https://en.wikipedia.org/wiki/A9.com#A9.com_search_portal

KMag 5y ago
Yes, A9 was one of the competitors for some of my time at Google, but I'm talking about Amazon basically creating some search engine infrastructure and a framework for building search engines on top of. Selling shovels to the gold miners, in a sense. The crawling and initial indexing stages need a lot of scaling and performance tuning work, which is a core competency for the AWS engineers. Furthermore, essentially zero people say "I want to build a startup around my new crawl scheduling idea, but it's too time consuming and expensive to write the rest of the search engine." Many more people say "I have a new idea for ranking" or "I have a new idea for content extraction, but it's too expensive and time-consuming to write the rest of a search engine". Google and Bing already expose "custom search engines" which are basically just filters on top of their search engines. I'm talking about something similar, but much deeper customization. Also, I was at Goldman when Google Search Appliance was EoL'd. GS had plenty of internal content that needed to be indexed, absolutely needed to be kept private, and wasn't well indexed by any of the replacements GS evaluated. For companies that keep lots of documents in AWS storage, it would be good to open up the space for Google Search Appliance replacements. Amazon could write their own, but given that Google abandoned the space, the profit margins aren't great for a huge company. However, creating a search ecosystem within the AWS ecosystem makes the AWS ecosystem more attractive. Maybe forgoing the search profits and inviting in search providers would be a net win in selling more AWS compute and storage.
- guyzero 5y ago
  Unfortunately I suspect this would be selling gold shovels to dirt miners and the economics of it are not compelling.

deleted 5y ago

[deleted]