4 ms·
You are implying web. If you eliminate web from your major components, the concepts are the same for building a search engine for any data. Like a product searc
by mattbrewsbytes 4y ago
You are implying web. If you eliminate web from your major components, the concepts are the same for building a search engine for any data. Like a product search on a company website might need to crawl through a bunch of internal data sources to build an index of all the products, metadata, etc. and you may want some products ranked higher in results for whatever reason.
I think a major effort, if its not web, is defining a common data structure to represent the varying source data sets.
Edit to add: OP if you want to study this, start smaller than "search the web" and get the major component concepts down. Expanding to the web will involve distributed systems stuff to scale but there's only like a handful of companies doing that yet there are many more companies that need a search engine on in-house data.
- oikawa_tooru 4y agoHi thank you for your answer. Actually my original idea was to build it for something else rather than the web in particular. Can you please advice me on which masters I should pick between distributed systems , machine learning and theoretical computer science/algorithm if my end goal is this?
- mattbrewsbytes 4y agoI don't think any specific MS is going to cover everything for building a search engine. You may want to look at people that work on Lucene [0], for example, and connect to see what their background might be. My guess is there is much more algorithms (types of searches and their big-O tradeoffs, sorts, etc.) involved than dist systems/ml but I don't work on search engines. Its open source so you could just download it and tinker. You may find that a degree isn't necessary to build one. [0] https://lucene.apache.org/ https://lucene.apache.org/