It's awesome, because companies use this to drive profit. I worked IT for a company that had an entire department (think up to ten people!) just crawling sites

It's awesome, because companies use this to drive profit. I worked IT for a company that had an entire department (think up to ten people!) just crawling sites like this all day to determine where to increase their advertising. They made flu-related medicines.