4 ms·
Message to pointy-haired executive: Google robots.txt Ironically, not only does the foxnews.com/robots.txt not block search engines, it has sitemaps for helpin
by va_coder 17y ago
Message to pointy-haired executive:
Google robots.txt
Ironically, not only does the foxnews.com/robots.txt not block search engines, it has sitemaps for helping them.
- wmeredith 17y agoExactly. I love all this blustering and ranting when all they have to do is install a text file in their home directory. That's it. It would literally take 5 minutes; maybe less. There is no cost of adoption to opting out of the search game. If there was any backbone behind all this rhetoric, they'd have already done it. But there isn't, they want to whine until Bing gives them a truckload of money for exclusivity or they get a government bailout. Dear news conglomerates, Please, install the robots.txt file to block search, which all reputable engines respect, and start suing the bloggers and aggregators. Please do this today, so you disappear from the net completely and the information void you leave behind can be filled by lean quality content producers who bitch less. Love, Wade
- igrekel 17y agoI don't see what the relationship should really be with robots.txt. Block their site will not make them profitable, it doesn't provide a source of revenue.
- jpwatts 17y agoThey claim to be generating the revenue, only to have it stolen by Google and others. If this was the case, /robots.txt could go a long way towards preventing billions of dollars from "leak[ing] onto the Internet" each year.
- igrekel 17y agoI had a different understanding of what the core point was, the google part felt more like a sub point. My understanding was that since the content is freely accessible on the internet, people don't need to subscribe to newspapers, and that online advertising can't cover for the difference.