6 ms·

MSNBOT must die

83 points by alrex021 17y ago

mrduncan 17y ago
I have to wonder if there have been some recent changes which are causing all kinds of issues. Just a few days ago, Ryan Tomayko twittered about it completely ignoring Github's robots.txt - http://twitter.com/rtomayko/statuses/7685967826 http://twitter.com/rtomayko/statuses/7685967826
- pjhyett 17y ago
  Our Mysql boxes were hammered thanks to this bug. Pages that weren't supposed to be crawled because they're resource intensive were all getting hit. Very frustrating, we ended up blocking all msnbot traffic in Nginx.
petercooper 17y ago
It sucks for them, but I doubt any of perl.org really needs to be indexed by Bing anyway (though it's always nice to be indexed). It's not an entirely fair or scientific comparison but my general Ruby site had ~55k visitors from Google in the last month. 280 from Bing. I'd suspect their market share in the Perl world is as bad or worse.
- jacquesm 17y ago
  The problem really is that every two bit search engine crawler (twiceler, yahoo, bing and so on) will use just as much bandwidth as google does (or more) but will only return a fraction of the number of users of google. Search engine vendors ought to do something like 80 legs and do something about crawling the web as inefficient as they do today.
  - OmarIsmail 17y ago
    Don't get me started on 80legs. They took down our servers earlier this week. We couldn't even identify them based on IP address because of their regular-computer-using distributed system. If anything, 80Legs is the ultimate DDoS machine. I mean, they state that their crawl average is 1 req/sec and can be modified if you contact them. The problem is that other people can modify the crawl rate (somebody set ours to 4/seq). And remember, that's just the average. The distributed nature of their system means that it's ridiculously spikey. Furthermore they don't respect nofollow tags that we put in place to keep bots out of bad areas, so their crawlers got into very unoptimized areas of the site. Overall though we turned it into a positive by ramping up our analysis tools, and creating a much stronger robots.txt. Still... I really don't like 80legs and bristle when people reference them as someone doing 'good'.
    - jdrock 17y ago
      So.. we usually get comments like this on webmaster forums, and I usually just let them go, but since HN is "my" forum, I feel like saying something. 1. We identify ourselves as user-agent 008. The proper way to track a bot is through user-agent, not IP address. 2. We never say that the average rate for all sites is 1 req/sec. Many sites have higher rates. We do a best guess on what load a site can handle based on a mix of Alexa/Quantcast/Compete stats. 3. We always respond to requests from webmasters to reduce the crawl rate. Within 5 minutes of getting your message, we did this. What other bot does this? Oh yeah, none of them. 4. The nofollow tag is not a standard way to tell bots to not go to a link. That is what robots.txt is for. Nofollow was implemented for Google specifically and has caught on as a (not as common as you think) tag. Robots.txt is the right way to talk to bots. 5. You know what happens when people don't use 80legs? They figure out some way to get the information they want and soak up your bandwidth anyway. A webmaster is fooling himself if he thinks he can stop people from crawling a site. It's going to happen. At least with 80legs the webmaster has a responsible company with real people to talk to so that the crawl can be managed.
      - petercooper 17y ago
        Perhaps the parent poster is getting confused by your wiki which states: 80legs defaults to an average rate limit of 1 page per domain per second, but this rate limit may be increased over time for certain domains. And then it goes on to mention people can contact you to increase the limit. This can be interpreted many ways, and the parent poster interpreted it in a different way to how you do.
      - OmarIsmail 17y ago
        1. This requires prior knowledge of your existence, it wasn't until we did proper analysis that we were able to identify the culprit of the "attack". This did expose a weakness in our monitoring systems, so we've obviously learned from this experience, but I can tell you that there are MANY webmasters on the web that aren't as savvy who can get their servers taken down easily by your service. 2. You should try reading your own FAQ. http://80legs.pbworks.com/FAQ http://80legs.pbworks.com/FAQ - see Rate limiting section. 3/4/5. The only bot I care about is Googlebot. And they do use nofollow that way, fortunately the other major bots do as well... so things have been working pretty darn great for everybody. Every other bot I don't care for, don't need, and don't want. If someone legit wants access to our data they can use our totally open and free API that actually gives a lot better information. In principal I think 80legs can be used for good, however, it's like gunpowder where I question the legitimacy of real-world practical use, simply because it can be abused so so easily. I could very well be totally off base here, but that's what lack of information + bad experience gets you. edit: And shouldn't the fact that you "...usually get comments like this on webmaster forums..." be a hint that maybe there's a real issue with your service?
aguynamedben 17y ago
Yep, MSNBOT does the same type of bursty crawling here too. Not graceful at all.
ecaron 17y ago
Their bots have just become incredibly greedy, while Google is becoming very tidy. Despite a growth of content on LinkUp.com, Google has gone from consuming 8gigs of my bandwidth in September 2009 to being on track to use only 4gigs this month. During those same time periods, MSNBot has gone from using 4gigs to using 15gigs this month. Sure Bing may be on a growth spurt (http://www.adotas.com/2010/01/bing-hits-query-growth-spurt/ http://www.adotas.com/2010/01/bing-hits-query-growth-spurt/), but I don't see any correlation between that and pwning our servers...
- jacquesm 17y ago
  Those two are not related, you could crawl all day long and never serve up a single query. Also, bing being on a 'growth spurt' is something that I can't find any hard proof for from other sources, They're pretty steady according to: http://www.alexa.com/siteinfo/bing.com http://www.alexa.com/siteinfo/bing.com and http://siteanalytics.compete.com/bing.com/ http://siteanalytics.compete.com/bing.com/
akkartik 17y ago
I mostly sympathize, but this is crazy: "I'll now be denying access to anything with the IP matching /^65\.55\.(106|107|207)/. If you discover you fall into that pattern, and are a real person, please let me know." How hard is it to lookup DNS?
- carbocation 17y ago
  I agree with your sentiment. My first thought, however, was that is extremely costly to dynamically lookup DNS for every page request. Even if a DNS lookup was only performed if the IP matched that range, you'd still end up doing a DNS lookup for every MSNBot hit. If they're calling way too many pages (which is the reason for the block in the first place), this will thrash your server. Now, if you followed the above but then added the IP to a static blacklist if it resolved to an MSFT host - and a whitelist if not - then the system should work without too much of a performance hit (except a small one upfront). Of course, at that rate, it would probably be easier to just submit a batch job to generate IPs in that range for blacklist and whitelist. Anyways, your rhetorical point still stands :-)
  - netdog 17y ago
    I just block that entire Microsoft subnet at the firewall (using pf): microsoft = "65.52.0.0/14" block in on $ext_if proto tcp from $microsoft to any port { http, https }
- ajross 17y ago
  It's actually extremely hard (in the performance sense) to do a reverse lookup of every inbound connection. Remote DNS servers can have terrible latency problems, and your framework needs to be prepared to have hundreds or thousands (or more, for a big site) of lookups in flight to avoid starving the valid connections. A netblock lookup is a single firewall rule.
  - akkartik 17y ago
    Just cache your lookups, perform them only for IPs in that region. It is seriously not that hard, there aren't that many MSN bots running around. It's weird that my comment above is getting downvoted when carbocation is agreeing with me and getting upvoted. This isn't downvoting by disagreement, so what did I say that was so wrong?
docgnome 17y ago
It's just straight up evil to run an indexing service that ignores robots.txt and to run more than one of these things that when they don't talk to each other. Even if the latter was stupidity, the former is just plain evil. It's not like they have an excuse of ignorance. This isn't some guy who just learned to program hacking something together. These are supposed to be "professional" programmers.
- chriseppstein 17y ago
  M$ never promised to not be evil.
- bockris 17y ago
  And it's nothing new. A couple of years ago I used to have a domain that I put a blanket disallow on. Live search indexed it but neither Yahoo or Google did.
bshep 17y ago
Why 403s instead of just denying the connection with hosts.deny? I'm thinking more bandwidth/CPU savings than sending correct responses for their crawler...
gojomo 17y ago
They should try to contact the crawling team directly. It's obviously a bug. While it's satisfying to fix with a block, or vent in a blog post or tweet, actually getting to the bottom of the issue with the MS crawl team can fix the problem for other sites, too -- so that's the most civil and community-spirited response.
- johnl 17y ago
  Yes, feedback is important. You can't test everything.
- rm-rf 17y ago
  Someone from MS reached out to the OP in a comment: "I am a Program Manager on the Bing team at Microsoft, thanks for bringing this issue to our attention." Have to give them credit for that.
- pierrefar 17y ago
  MSNBot has had many issues over the years. Many times MS folks denied them all until someone actually caused a big fuss. Once it was me causing the fuss. When you think about it, bing/live/whatever it's called today sends very little traffic, and the crawler to get that traffic has a long history of what you call bugs. The cost-benefit analysis IMHO points to too much cost for really zero benefit. Hence why many webmasters block MSNBot.

83 points by alrex021 17y ago

I have to wonder if there have been some recent changes which are causing all kinds of issues. Just a few days ago, Ryan Tomayko twittered about it completely ignoring Github's robots.txt - http://twitter.com/rtomayko/statuses/7685967826 http://twitter.com/rtomayko/statuses/7685967826

pjhyett 17y ago
Our Mysql boxes were hammered thanks to this bug. Pages that weren't supposed to be crawled because they're resource intensive were all getting hit. Very frustrating, we ended up blocking all msnbot traffic in Nginx.

petercooper 17y ago

It sucks for them, but I doubt any of perl.org really needs to be indexed by Bing anyway (though it's always nice to be indexed). It's not an entirely fair or scientific comparison but my general Ruby site had ~55k visitors from Google in the last month. 280 from Bing. I'd suspect their market share in the Perl world is as bad or worse.

jacquesm 17y ago
The problem really is that every two bit search engine crawler (twiceler, yahoo, bing and so on) will use just as much bandwidth as google does (or more) but will only return a fraction of the number of users of google. Search engine vendors ought to do something like 80 legs and do something about crawling the web as inefficient as they do today.
- OmarIsmail 17y ago
  Don't get me started on 80legs. They took down our servers earlier this week. We couldn't even identify them based on IP address because of their regular-computer-using distributed system. If anything, 80Legs is the ultimate DDoS machine. I mean, they state that their crawl average is 1 req/sec and can be modified if you contact them. The problem is that other people can modify the crawl rate (somebody set ours to 4/seq). And remember, that's just the average. The distributed nature of their system means that it's ridiculously spikey. Furthermore they don't respect nofollow tags that we put in place to keep bots out of bad areas, so their crawlers got into very unoptimized areas of the site. Overall though we turned it into a positive by ramping up our analysis tools, and creating a much stronger robots.txt. Still... I really don't like 80legs and bristle when people reference them as someone doing 'good'.
  - jdrock 17y ago
    So.. we usually get comments like this on webmaster forums, and I usually just let them go, but since HN is "my" forum, I feel like saying something. 1. We identify ourselves as user-agent 008. The proper way to track a bot is through user-agent, not IP address. 2. We never say that the average rate for all sites is 1 req/sec. Many sites have higher rates. We do a best guess on what load a site can handle based on a mix of Alexa/Quantcast/Compete stats. 3. We always respond to requests from webmasters to reduce the crawl rate. Within 5 minutes of getting your message, we did this. What other bot does this? Oh yeah, none of them. 4. The nofollow tag is not a standard way to tell bots to not go to a link. That is what robots.txt is for. Nofollow was implemented for Google specifically and has caught on as a (not as common as you think) tag. Robots.txt is the right way to talk to bots. 5. You know what happens when people don't use 80legs? They figure out some way to get the information they want and soak up your bandwidth anyway. A webmaster is fooling himself if he thinks he can stop people from crawling a site. It's going to happen. At least with 80legs the webmaster has a responsible company with real people to talk to so that the crawl can be managed.
    - petercooper 17y ago
      Perhaps the parent poster is getting confused by your wiki which states: 80legs defaults to an average rate limit of 1 page per domain per second, but this rate limit may be increased over time for certain domains. And then it goes on to mention people can contact you to increase the limit. This can be interpreted many ways, and the parent poster interpreted it in a different way to how you do.
    - OmarIsmail 17y ago
      1. This requires prior knowledge of your existence, it wasn't until we did proper analysis that we were able to identify the culprit of the "attack". This did expose a weakness in our monitoring systems, so we've obviously learned from this experience, but I can tell you that there are MANY webmasters on the web that aren't as savvy who can get their servers taken down easily by your service. 2. You should try reading your own FAQ. http://80legs.pbworks.com/FAQ http://80legs.pbworks.com/FAQ - see Rate limiting section. 3/4/5. The only bot I care about is Googlebot. And they do use nofollow that way, fortunately the other major bots do as well... so things have been working pretty darn great for everybody. Every other bot I don't care for, don't need, and don't want. If someone legit wants access to our data they can use our totally open and free API that actually gives a lot better information. In principal I think 80legs can be used for good, however, it's like gunpowder where I question the legitimacy of real-world practical use, simply because it can be abused so so easily. I could very well be totally off base here, but that's what lack of information + bad experience gets you. edit: And shouldn't the fact that you "...usually get comments like this on webmaster forums..." be a hint that maybe there's a real issue with your service?

aguynamedben 17y ago

Yep, MSNBOT does the same type of bursty crawling here too. Not graceful at all.

ecaron 17y ago

Their bots have just become incredibly greedy, while Google is becoming very tidy. Despite a growth of content on LinkUp.com, Google has gone from consuming 8gigs of my bandwidth in September 2009 to being on track to use only 4gigs this month. During those same time periods, MSNBot has gone from using 4gigs to using 15gigs this month. Sure Bing may be on a growth spurt (http://www.adotas.com/2010/01/bing-hits-query-growth-spurt/ http://www.adotas.com/2010/01/bing-hits-query-growth-spurt/), but I don't see any correlation between that and pwning our servers...

jacquesm 17y ago
Those two are not related, you could crawl all day long and never serve up a single query. Also, bing being on a 'growth spurt' is something that I can't find any hard proof for from other sources, They're pretty steady according to: http://www.alexa.com/siteinfo/bing.com http://www.alexa.com/siteinfo/bing.com and http://siteanalytics.compete.com/bing.com/ http://siteanalytics.compete.com/bing.com/

akkartik 17y ago

I mostly sympathize, but this is crazy: "I'll now be denying access to anything with the IP matching /^65\.55\.(106|107|207)/. If you discover you fall into that pattern, and are a real person, please let me know." How hard is it to lookup DNS?

carbocation 17y ago
I agree with your sentiment. My first thought, however, was that is extremely costly to dynamically lookup DNS for every page request. Even if a DNS lookup was only performed if the IP matched that range, you'd still end up doing a DNS lookup for every MSNBot hit. If they're calling way too many pages (which is the reason for the block in the first place), this will thrash your server. Now, if you followed the above but then added the IP to a static blacklist if it resolved to an MSFT host - and a whitelist if not - then the system should work without too much of a performance hit (except a small one upfront). Of course, at that rate, it would probably be easier to just submit a batch job to generate IPs in that range for blacklist and whitelist. Anyways, your rhetorical point still stands :-)
- netdog 17y ago
  I just block that entire Microsoft subnet at the firewall (using pf): microsoft = "65.52.0.0/14" block in on $ext_if proto tcp from $microsoft to any port { http, https }
ajross 17y ago
It's actually extremely hard (in the performance sense) to do a reverse lookup of every inbound connection. Remote DNS servers can have terrible latency problems, and your framework needs to be prepared to have hundreds or thousands (or more, for a big site) of lookups in flight to avoid starving the valid connections. A netblock lookup is a single firewall rule.
- akkartik 17y ago
  Just cache your lookups, perform them only for IPs in that region. It is seriously not that hard, there aren't that many MSN bots running around. It's weird that my comment above is getting downvoted when carbocation is agreeing with me and getting upvoted. This isn't downvoting by disagreement, so what did I say that was so wrong?

docgnome 17y ago

It's just straight up evil to run an indexing service that ignores robots.txt and to run more than one of these things that when they don't talk to each other. Even if the latter was stupidity, the former is just plain evil. It's not like they have an excuse of ignorance. This isn't some guy who just learned to program hacking something together. These are supposed to be "professional" programmers.

chriseppstein 17y ago
M$ never promised to not be evil.
bockris 17y ago
And it's nothing new. A couple of years ago I used to have a domain that I put a blanket disallow on. Live search indexed it but neither Yahoo or Google did.

bshep 17y ago

Why 403s instead of just denying the connection with hosts.deny? I'm thinking more bandwidth/CPU savings than sending correct responses for their crawler...

gojomo 17y ago

They should try to contact the crawling team directly. It's obviously a bug. While it's satisfying to fix with a block, or vent in a blog post or tweet, actually getting to the bottom of the issue with the MS crawl team can fix the problem for other sites, too -- so that's the most civil and community-spirited response.

johnl 17y ago
Yes, feedback is important. You can't test everything.
rm-rf 17y ago
Someone from MS reached out to the OP in a comment: "I am a Program Manager on the Bing team at Microsoft, thanks for bringing this issue to our attention." Have to give them credit for that.
pierrefar 17y ago
MSNBot has had many issues over the years. Many times MS folks denied them all until someone actually caused a big fuss. Once it was me causing the fuss. When you think about it, bing/live/whatever it's called today sends very little traffic, and the crawler to get that traffic has a long history of what you call bugs. The cost-benefit analysis IMHO points to too much cost for really zero benefit. Hence why many webmasters block MSNBot.