3 ms·
And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms,
by sjwright 10y ago
And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.
- greglindahl 10y agoA majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
- sjwright 10y agoI agree, it's a difficult conundrum. It sucks.
- greglindahl 10y agoThere are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...