I'd identify them and then deny them. To identify them, I'd use a few tricks. One would be to place invisible links on some of your pages. These could be empty

I'd identify them and then deny them. To identify them, I'd use a few tricks. One would be to place invisible links on some of your pages. These could be empty anchor tags, or hidden by javascript so that normal users don't click on them. If someone clicks on them, they're probably a bot or something odd. Give them a particular cookie, or mark their IP address as problematic. Another way to identify them is to look at browsing behavior such as requests per minute, browser name, not fetching images, and the "other patterns in their requests" that you mention. Denying them is probably easier. Put in an entry for their IP in iptables, or your routing table, or use an apache module to serve them something that doesn't require a lot of computing power. Deny them for some period of time, like a few minutes or hours. You're right that you can offer to trade data with them directly. Build an API and ask them to use it. If it's easier for them, they'll do it. I'm not sure if this undermines your business or not without knowing more details. If you don't know who the competitor is, put some bogus items on your site. Let them crawl them and put them on their site. Then, you can search Google for those oddly named items. Anyone who has them on their site has been gathering data from you. This is an arms race. You can't really stop it, but you can make the desired behavior easier for them.