3 ms·
(On that note) Fun fact, there are a lot of sites out there that have User-Agent: * Disallow / User-Agent: Googlebot Allow / -- and yet those
by rezonant 4y ago
(On that note) Fun fact, there are a lot of sites out there that have
User-Agent: *
Disallow /
User-Agent: Googlebot
Allow /
-- and yet those sites are in the Bing index.
Do I blame Bing for that? Absolutely not.
Having a silly robots.txt is one thing- going to the lengths of blocking all bots that aren't in Google's IP space is what I'm worried about. If the whole world did that, nothing but Google could survive.
- shagie 4y agoThose are absolutely problems. I suspect they're artifacts in configurations from when it was believed that Google did no evil and the next biggest company trying to get in on the web search was Microsoft and leaving them in wasn't a great loss. Now roles are reversed (or at least, equal - though if DDG consuming bing data is considered to be on the side of good, an argument could be made for reversed) and yet those configurations persist because they were forgotten about. My post above is a just so story ( https://en.wikipedia.org/wiki/Just-so_story https://en.wikipedia.org/wiki/Just-so_story ) for why they are the way they are now. Old systems that work tend to be forgotten until a security fix needs to go in and Chesterton's fences in config files tend to be left. With todays resources, I would not advocate indiscriminate tcp tarpits for things that might be badly behaved crawlers until they demonstrate that they are indeed badly behaved crawlers or scanners searching for exploits.
- brazzledazzle 4y agoUnless I missed or forgot some major events I don’t think it’s fair to say that Microsoft and Google are on the same level of “evil”. Microsoft willfully broke the law to destroy their competition. On multiple occasions. They had well funded initiatives to destroy Linux and open source in general. They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. Microsoft was (and is) ruthless because Bill Gates was a fucking corporate terminator.
- saagarjha 4y ago> They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. …something which Google has never done, I'm sure.
- robertlagrant 4y agoGoogle seems to have done much more in the open source and standards space. E.g. buying and open sourcing VP8.
- fsckboy 4y agoall monopolies are bad by their very nature. If consumers have no choice, they get abused, regardless of whether it is clumsy and barbaric like microsoft, or deft and charming like Hannibal Google Lector.
- rezonant 4y agoUnfortunately, these are major, modern websites. I'm not talking about mom and pop sites here :-/
- akerl_ 4y agoDid you comment this in the wrong place? It doesn't seem to be a response to what I said. shagie brought up the possibility that somebody would have a robots.txt with only Google in it because when they wrote their robots.txt, they wanted to block poorly behaved / evil crawlers and Google's was well-behaved. But that doesn't make sense to me, because by definition poorly behaved / evil crawlers do not care about robots.txt.
- shagie 4y agoThe ones that weren't well behaved back then found themselves in a tarpit (which was also a 'what you do with spam email sites connecting on port 25'). https://en.wikipedia.org/wiki/Tarpit_(networking) https://en.wikipedia.org/wiki/Tarpit_(networking) The thing was that you'd send everything that was requesting faster than some rate limit to that tarpit unless it was google.
- rezonant 4y agoRiffing off what you wrote, not a rebuttal.