4 ms·

That's what I'm thinking as well. I only need to do a crawler, doing HEAD requests only, piping into /dev/null, in theory.

• vsgxvhdxh 10y ago
  So all your head requests get filtered out when the db gets queried. Leaving all your gets and posts etc. Also better hope your crawler doesn't hit any illegal sites (there are loads according to UK law).
  • DanBC 10y ago
    > (there are loads according to UK law). No, there really aren't.