11 ms·

I mostly sympathize, but this is crazy: "I'll now be denying access to anything with the IP matching /^65\.55\.(106|107|207)/. If you discover you fall into th

I mostly sympathize, but this is crazy: "I'll now be denying access to anything with the IP matching /^65\.55\.(106|107|207)/. If you discover you fall into that pattern, and are a real person, please let me know." How hard is it to lookup DNS?

carbocation 17y ago
I agree with your sentiment. My first thought, however, was that is extremely costly to dynamically lookup DNS for every page request. Even if a DNS lookup was only performed if the IP matched that range, you'd still end up doing a DNS lookup for every MSNBot hit. If they're calling way too many pages (which is the reason for the block in the first place), this will thrash your server. Now, if you followed the above but then added the IP to a static blacklist if it resolved to an MSFT host - and a whitelist if not - then the system should work without too much of a performance hit (except a small one upfront). Of course, at that rate, it would probably be easier to just submit a batch job to generate IPs in that range for blacklist and whitelist. Anyways, your rhetorical point still stands :-)
- netdog 17y ago
  I just block that entire Microsoft subnet at the firewall (using pf): microsoft = "65.52.0.0/14" block in on $ext_if proto tcp from $microsoft to any port { http, https }
ajross 17y ago
It's actually extremely hard (in the performance sense) to do a reverse lookup of every inbound connection. Remote DNS servers can have terrible latency problems, and your framework needs to be prepared to have hundreds or thousands (or more, for a big site) of lookups in flight to avoid starving the valid connections. A netblock lookup is a single firewall rule.
- akkartik 17y ago
  Just cache your lookups, perform them only for IPs in that region. It is seriously not that hard, there aren't that many MSN bots running around. It's weird that my comment above is getting downvoted when carbocation is agreeing with me and getting upvoted. This isn't downvoting by disagreement, so what did I say that was so wrong?

carbocation 17y ago

I agree with your sentiment. My first thought, however, was that is extremely costly to dynamically lookup DNS for every page request. Even if a DNS lookup was only performed if the IP matched that range, you'd still end up doing a DNS lookup for every MSNBot hit. If they're calling way too many pages (which is the reason for the block in the first place), this will thrash your server. Now, if you followed the above but then added the IP to a static blacklist if it resolved to an MSFT host - and a whitelist if not - then the system should work without too much of a performance hit (except a small one upfront). Of course, at that rate, it would probably be easier to just submit a batch job to generate IPs in that range for blacklist and whitelist. Anyways, your rhetorical point still stands :-)

netdog 17y ago
I just block that entire Microsoft subnet at the firewall (using pf): microsoft = "65.52.0.0/14" block in on $ext_if proto tcp from $microsoft to any port { http, https }

ajross 17y ago

It's actually extremely hard (in the performance sense) to do a reverse lookup of every inbound connection. Remote DNS servers can have terrible latency problems, and your framework needs to be prepared to have hundreds or thousands (or more, for a big site) of lookups in flight to avoid starving the valid connections. A netblock lookup is a single firewall rule.

akkartik 17y ago
Just cache your lookups, perform them only for IPs in that region. It is seriously not that hard, there aren't that many MSN bots running around. It's weird that my comment above is getting downvoted when carbocation is agreeing with me and getting upvoted. This isn't downvoting by disagreement, so what did I say that was so wrong?