Write a crawler: Make it nice, so it doesn't hammer sites, convert relative urls to absolute, pull out relative data from html, cycle urls back through crawler

Write a crawler: Make it nice, so it doesn't hammer sites, convert relative urls to absolute, pull out relative data from html, cycle urls back through crawler for more crawling. Make sure your ISP doesn't have a cap on your monthly bandwidth (i.e. don't use a residential-scale ISP). Take into account pages that are dupes of other pages, so compare MD5s of all content to detect urls that generate duplicate content to what you've already pulled and ignore those. Integrate sitemap detection for smarter crawls. Index the data: Figure out a way to index the massive volume of data that's not a live query system (lucene starts to gag at queries > 50GB or so), so generate all possible results for all possible queries in a database and update those results occasionally as you pull in more data. When someone queries for a search string, pull up the results from a pre-generated query from a database, don't do a live search of all of your terrabytes of data, or the query will take days. Generate interesting results: find a niche. Don't plan to take on Google, Bing & Yahoo on a personal scale. People put together good engines, but target main sites, and front pages, or a shallow-depth crawl. Don't plan on indexing every forum and every blog on the internet. I've been impressed by this guy's search engine: http://gigablast.com/ http://gigablast.com/ In short, unless you've got a TON of money, machinery, people and time, don't try to compete with Google. Find a niche like shopping search or movie search, or be human-powered like Mahalo. Google's got a dedicated computer for every possible search query out there or close to it, plus a team of 500,000 Chinese people making sure that popular results are relevant (Google does human-validated results for many of the most popular queries, not like an error message query).