I posted this a few years back, problem has since become only more difficult: https://news.ycombinator.com/item?id=11626999 https://news.ycombinator.com/item?id

I posted this a few years back, problem has since become only more difficult: https://news.ycombinator.com/item?id=11626999 https://news.ycombinator.com/item?id=11626999 Writing an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in: * a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicating this in a distributed manner will be played politically * Assuming any significant usage, the second you submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage * Executing custom ranking function on other people's computer not only introduces security risks, but will have scammers setting up honeypots for collecting other people's ranking signals, and playing accordingly.