There's absolutely no reason. Then again, there's plety of bad code to go around in this crawler. Just a few nuggets: * Magic strings with no comments. * Gl

There's absolutely no reason. Then again, there's plety of bad code to go around in this crawler. Just a few nuggets: * Magic strings with no comments. * Global variables * Attempting to write his own URI parser.