17 ms·

Agree with the suit but QVC (by this time) should have rate limiting / throttling per IP. (waits for somebody to claim that each request came from a different

by korzun 12y ago

Agree with the suit but QVC (by this time) should have rate limiting / throttling per IP. (waits for somebody to claim that each request came from a different proxy)

greglindahl 12y ago
From TFA: The complaint alleges that the defendant disguised its web crawler to mask its source IP address and thus prevented QVC technicians from identifying the source of the requests and quickly repairing the problem.
- korzun 12y ago
  Yes. Did your read/understand what I posted? Especially the last part where I predicted that somebody will crawl out and post what you just did?
atwebb 12y ago
> (waits for somebody to claim that each request came from a different proxy) Seems they were using proxies but you don't want to go blocking IPs making a number of requests, right? they could be legitmate shoppers opening lots of tabs, refreshing, hitting image/resource heavy pages, analytics etc. I'm not a network or server admin though so maybe there are some tools or methods that help identify the bad traffic.
- korzun 12y ago
  Rate limiting is not blocking. Multiple tabs do not translate to concurrent requests. A shop like QVC should have a high limit and frankly an automatic system that flags stuff like this and notifies proper parties.
  - atwebb 12y ago
    Yep, I misread that one as blocking/denying.
chadgeidel 12y ago
This happened to us recently (however not at this scale), and I assure you this wouldn't solve anything. We only got a few requests per IP.
- korzun 12y ago
  I assure you that if you had a proper system in place it would limit the exposure. We are talking about 600 r/s (QVC should be able to eat that anyways, but whatever). At this level you should know this will happen and have placeholders in place. If every single IP is unique, additional layer would be pattern matching traffic by different network classes and flagging it actively. Another way to rate limit is by number of pages per session / time. Then you can have another fail safe method where if X exceeds Y start throttling top X (dynamically scale up until X is returned to normal) while notifying DevOps of the issue. Etc.
  - chadgeidel 12y ago
    Your initial comment said "but QVC (by this time) should have rate limiting / throttling per IP." If you have 1 or 2 requests from an IP then rate limiting those requests does nothing. Your other suggestions are certainly interesting, and you may have implied those solutions - but that's not what you originally said. You said rate limiting by IP is enough. It is not.
    - korzun 12y ago
      Fair point.

by korzun 12y ago

Agree with the suit but QVC (by this time) should have rate limiting / throttling per IP. (waits for somebody to claim that each request came from a different proxy)

greglindahl 12y ago

From TFA: The complaint alleges that the defendant disguised its web crawler to mask its source IP address and thus prevented QVC technicians from identifying the source of the requests and quickly repairing the problem.

korzun 12y ago
Yes. Did your read/understand what I posted? Especially the last part where I predicted that somebody will crawl out and post what you just did?

atwebb 12y ago

> (waits for somebody to claim that each request came from a different proxy) Seems they were using proxies but you don't want to go blocking IPs making a number of requests, right? they could be legitmate shoppers opening lots of tabs, refreshing, hitting image/resource heavy pages, analytics etc. I'm not a network or server admin though so maybe there are some tools or methods that help identify the bad traffic.

korzun 12y ago
Rate limiting is not blocking. Multiple tabs do not translate to concurrent requests. A shop like QVC should have a high limit and frankly an automatic system that flags stuff like this and notifies proper parties.
- atwebb 12y ago
  Yep, I misread that one as blocking/denying.

chadgeidel 12y ago

This happened to us recently (however not at this scale), and I assure you this wouldn't solve anything. We only got a few requests per IP.

korzun 12y ago
I assure you that if you had a proper system in place it would limit the exposure. We are talking about 600 r/s (QVC should be able to eat that anyways, but whatever). At this level you should know this will happen and have placeholders in place. If every single IP is unique, additional layer would be pattern matching traffic by different network classes and flagging it actively. Another way to rate limit is by number of pages per session / time. Then you can have another fail safe method where if X exceeds Y start throttling top X (dynamically scale up until X is returned to normal) while notifying DevOps of the issue. Etc.
- chadgeidel 12y ago
  Your initial comment said "but QVC (by this time) should have rate limiting / throttling per IP." If you have 1 or 2 requests from an IP then rate limiting those requests does nothing. Your other suggestions are certainly interesting, and you may have implied those solutions - but that's not what you originally said. You said rate limiting by IP is enough. It is not.
  - korzun 12y ago
    Fair point.