4 ms·
One issue with that is that there has been an arms race since 2008 with "search engine optimization". If you used a crawler from 2008 on today's web, you'd prob
by windsurfer 7y ago
One issue with that is that there has been an arms race since 2008 with "search engine optimization". If you used a crawler from 2008 on today's web, you'd probably get a ton of spam.
- codingslave 7y agoThe results for half of my searches are already all spam. It would be better to just accept that and have google let me engineer my queries so that I avoid spam. More token based searches, less word vector/machine learning based search. Let me query their index like its an SQL database
- windsurfer 7y agoAre you sure they are spam? Google's engineers consider annoying things like recipe pages with 20 paragraphs of stories before the actual recipe not to be spam.
- buboard 7y agoi am being tricked into pinterest link-pages multiple times a day. That is spam ,and has very little to do with their AI work
- im3w1l 7y agoRestricting it to 2008's web could be an interesting exercise.
- drusepth 7y agoI'm now super curious what percentage of pages from 2008 are still around/available today.