4 ms·

Result.ly are really a bunch of jerks. One of the most common sense things you can possibly do while crawling a website is monitor the response time and/or err

by birken 12y ago

Result.ly are really a bunch of jerks. One of the most common sense things you can possibly do while crawling a website is monitor the response time and/or error rates from the sites you are crawling. If those are going up, your crawl rate should go down or go to 0. There is one form of internet justice, which is QVC should file abuse complaints to the ISPs that host those IPs. I've found abuse complaints are the best way to stop people from using IPs for bad activities (excessive scraping, spamming, etc).

greglindahl 12y ago
From the complaint, it seems that Result.ly was crawling through proxies... which means QVC doesn't know whom to complain to: The complaint alleges that the defendant disguised its web crawler to mask its source IP address and thus prevented QVC technicians from identifying the source of the requests and quickly repairing the problem. Your comment about crawler politeness is spot-on.
- birken 12y ago
  My reading of that is they were splitting their crawling amongst a large block of IPs they had. If they are using proxies, that is much easier because you can just block them all without any worry of accidentally blocking real consumers (in addition to the fact that you can also file abuse complaints). I think at Thumbtack we blocked all AWS IPs, a good deal of foreign IPs, in addition to the specific IPs of people who were abusively crawling. At the same time, the onus shouldn't be on QVC to have to block this, result.ly should either be a good citizen or have to face a lawsuit. Granted, QVC's tech team should be able to deal with this because next time the person who is DOSing them might not be a US entity which can be sued, but that isn't entirely relevant in this situation.
  - greglindahl 12y ago
    How do you block a proxy network where each IP makes only 1 request? I've seen a lot of that at blekko, we're a big target for scrapers. I have no idea if that's what was happening to QVC, given the non-technical nature of the article.
    - birken 12y ago
      You likely have more experience with this than I do, but it seems to me that getting access to 36,000 IPs (the quoted maximum amount per minute) to make a single request each would be extremely difficult. And even if you could do that, most of them would probably be in similar blocks which would make them easier to detect and stop. It just is really hard and/or really expensive to get access to that many IPs (short of running a large botnet, which also would be hard and expensive and illegal). Based on QVC's inability to stop this I think their tech competence is probably fairly low, so my guess is that result.ly just would spin up 100 AWS instances, crawl for some amount of time (with a spoofed user agent), then close those down and spin up another 100 instances, then rinse and repeat. For QVC it would seem like you are constantly facing a moving target with all these different IPs, but in fact it is just from AWS. The complaint is of course inaccurate in that it is impossible to lie about the source of the request. Whatever IP the request was coming from is the IP it was coming from. Perhaps in the case of an AWS instance Result.ly is only leasing it for a short time, or in the case of a proxy it is the proxy's IP, but obviously it is still traceable to the owner somehow.
      - greglindahl 12y ago
        Yes, I have seen botnets which are >> 36,000 IPs. Again, from the lack of technical information, I have no idea what actually happened to QVC.

by birken 12y ago

greglindahl 12y ago

From the complaint, it seems that Result.ly was crawling through proxies... which means QVC doesn't know whom to complain to: The complaint alleges that the defendant disguised its web crawler to mask its source IP address and thus prevented QVC technicians from identifying the source of the requests and quickly repairing the problem. Your comment about crawler politeness is spot-on.

birken 12y ago
My reading of that is they were splitting their crawling amongst a large block of IPs they had. If they are using proxies, that is much easier because you can just block them all without any worry of accidentally blocking real consumers (in addition to the fact that you can also file abuse complaints). I think at Thumbtack we blocked all AWS IPs, a good deal of foreign IPs, in addition to the specific IPs of people who were abusively crawling. At the same time, the onus shouldn't be on QVC to have to block this, result.ly should either be a good citizen or have to face a lawsuit. Granted, QVC's tech team should be able to deal with this because next time the person who is DOSing them might not be a US entity which can be sued, but that isn't entirely relevant in this situation.
- greglindahl 12y ago
  How do you block a proxy network where each IP makes only 1 request? I've seen a lot of that at blekko, we're a big target for scrapers. I have no idea if that's what was happening to QVC, given the non-technical nature of the article.
  - birken 12y ago
    You likely have more experience with this than I do, but it seems to me that getting access to 36,000 IPs (the quoted maximum amount per minute) to make a single request each would be extremely difficult. And even if you could do that, most of them would probably be in similar blocks which would make them easier to detect and stop. It just is really hard and/or really expensive to get access to that many IPs (short of running a large botnet, which also would be hard and expensive and illegal). Based on QVC's inability to stop this I think their tech competence is probably fairly low, so my guess is that result.ly just would spin up 100 AWS instances, crawl for some amount of time (with a spoofed user agent), then close those down and spin up another 100 instances, then rinse and repeat. For QVC it would seem like you are constantly facing a moving target with all these different IPs, but in fact it is just from AWS. The complaint is of course inaccurate in that it is impossible to lie about the source of the request. Whatever IP the request was coming from is the IP it was coming from. Perhaps in the case of an AWS instance Result.ly is only leasing it for a short time, or in the case of a proxy it is the proxy's IP, but obviously it is still traceable to the owner somehow.
    - greglindahl 12y ago
      Yes, I have seen botnets which are >> 36,000 IPs. Again, from the lack of technical information, I have no idea what actually happened to QVC.