7 ms·
A crawler should see the web pages as a human would and read the pages as a human would and extract keywords based on the real displayed content. After that the
by fspeech 4y ago
A crawler should see the web pages as a human would and read the pages as a human would and extract keywords based on the real displayed content. After that the search could be as simple as looking up a book in your library catalog. Think liberally about the search key, e.g. "<40% screen is display ads". If the search engine gets popular, you can then worry about scaling up to serve the results faster.
Oh PageRank (R) can be done using real semantic links (actual human comments) rather than easily spammed hyperlinks. And you can group all similar pages and serve those as a single expandable/refinable entry.
- alkonaut 4y agoYes, telling spam (spam blogs, copy-pasted versions of Stackoverflow, etc) from real organic content is easy for a human, so I can only assume it would be pretty easy for a computer too. At least until this becomes a practice that spam site makers have to consider and adapt. For example, given 10 similar articles where one has a well-known domain but few links and the others are all cross-linked only among a known spam-net, then just show the well known page (stackoverflow) and not the useless copycats. It seems doable for a computer to also navigate the page like a human and try to negotiate various cookiewalls and similar. If a page requires jumping through hoops to enter, rank it poorly. There should be a single decline/reject if there is a button at all, and the content ranked should be the content shown after rejecting. If it shows two buttons "Accept" and "Options" then just don't index that page.
- denton-scratch 4y ago> If a page requires jumping through hoops to enter, rank it poorly. Oh, how I wish that search engines would simply give up on encountering a paywall/cookiewall, and refuse to index the page.