We fucked up by not vetting the algorithm, that's definitely the primary lesson here. Mea culpa. Bingo. Exactly. I heavily criticized you in another post for

We fucked up by not vetting the algorithm, that's definitely the primary lesson here. Mea culpa. Bingo. Exactly. I heavily criticized you in another post for 1) your initial choice of substandard algorithm and 2) your rationalizations (e.g. Google's "good reputation"). But that is ancient history. Here you're admitting that you fucked up, which IMO you didn't admit previously. Also, kudos for your article which kicked off this entire discussion. In that article you showed that you carefully analyzed what was happening, how you went wrong, and how you could improve. Even more important, you did this all publicly, both your article and your responses here on HN. Everyone learns from this. I commend you for your openness.