3 ms·
A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
by greglindahl 10y ago
A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
- sjwright 10y agoI agree, it's a difficult conundrum. It sucks.
- greglindahl 10y agoThere are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...