3 ms·
Wanted to say congrats on launching! I'm building a search engine myself, I can tell a lot of work went into this. I think the biggest thing you overlooked are
by gregw134 3y ago
Wanted to say congrats on launching! I'm building a search engine myself, I can tell a lot of work went into this.
I think the biggest thing you overlooked are page titles. When you issue a query it's a bit hard to quickly scan and judge what a site is about because the page titles are missing.
- pooper 3y agoHow do you crawl the web? Do you follow links around? How do you reach a page that isn't linked from anywhere you've crawled?
- gregw134 3y agoI'm just using common crawl for now
- mmkos 3y agoI mean that's what web crawling is, right? By extension, you just can't reach a page unless you stumble upon a link to it _somewhere_. Google gives you an option to submit a link and schedule a crawl that way, so that's another option if it's not being linked to from anywhere.