3 ms·

My reading of that is they were splitting their crawling amongst a large block of IPs they had. If they are using proxies, that is much easier because you can

by birken 12y ago

My reading of that is they were splitting their crawling amongst a large block of IPs they had. If they are using proxies, that is much easier because you can just block them all without any worry of accidentally blocking real consumers (in addition to the fact that you can also file abuse complaints). I think at Thumbtack we blocked all AWS IPs, a good deal of foreign IPs, in addition to the specific IPs of people who were abusively crawling. At the same time, the onus shouldn't be on QVC to have to block this, result.ly should either be a good citizen or have to face a lawsuit. Granted, QVC's tech team should be able to deal with this because next time the person who is DOSing them might not be a US entity which can be sued, but that isn't entirely relevant in this situation.

greglindahl 12y ago
How do you block a proxy network where each IP makes only 1 request? I've seen a lot of that at blekko, we're a big target for scrapers. I have no idea if that's what was happening to QVC, given the non-technical nature of the article.
- birken 12y ago
  You likely have more experience with this than I do, but it seems to me that getting access to 36,000 IPs (the quoted maximum amount per minute) to make a single request each would be extremely difficult. And even if you could do that, most of them would probably be in similar blocks which would make them easier to detect and stop. It just is really hard and/or really expensive to get access to that many IPs (short of running a large botnet, which also would be hard and expensive and illegal). Based on QVC's inability to stop this I think their tech competence is probably fairly low, so my guess is that result.ly just would spin up 100 AWS instances, crawl for some amount of time (with a spoofed user agent), then close those down and spin up another 100 instances, then rinse and repeat. For QVC it would seem like you are constantly facing a moving target with all these different IPs, but in fact it is just from AWS. The complaint is of course inaccurate in that it is impossible to lie about the source of the request. Whatever IP the request was coming from is the IP it was coming from. Perhaps in the case of an AWS instance Result.ly is only leasing it for a short time, or in the case of a proxy it is the proxy's IP, but obviously it is still traceable to the owner somehow.
  - greglindahl 12y ago
    Yes, I have seen botnets which are >> 36,000 IPs. Again, from the lack of technical information, I have no idea what actually happened to QVC.

by birken 12y ago

greglindahl 12y ago

How do you block a proxy network where each IP makes only 1 request? I've seen a lot of that at blekko, we're a big target for scrapers. I have no idea if that's what was happening to QVC, given the non-technical nature of the article.

birken 12y ago
You likely have more experience with this than I do, but it seems to me that getting access to 36,000 IPs (the quoted maximum amount per minute) to make a single request each would be extremely difficult. And even if you could do that, most of them would probably be in similar blocks which would make them easier to detect and stop. It just is really hard and/or really expensive to get access to that many IPs (short of running a large botnet, which also would be hard and expensive and illegal). Based on QVC's inability to stop this I think their tech competence is probably fairly low, so my guess is that result.ly just would spin up 100 AWS instances, crawl for some amount of time (with a spoofed user agent), then close those down and spin up another 100 instances, then rinse and repeat. For QVC it would seem like you are constantly facing a moving target with all these different IPs, but in fact it is just from AWS. The complaint is of course inaccurate in that it is impossible to lie about the source of the request. Whatever IP the request was coming from is the IP it was coming from. Perhaps in the case of an AWS instance Result.ly is only leasing it for a short time, or in the case of a proxy it is the proxy's IP, but obviously it is still traceable to the owner somehow.
- greglindahl 12y ago
  Yes, I have seen botnets which are >> 36,000 IPs. Again, from the lack of technical information, I have no idea what actually happened to QVC.