4 ms·

I enjoyed the article and would recommend it. Aside: At the end of the article they have a "Bonus: Speeding things up" section where they automate adding 300~

I enjoyed the article and would recommend it. Aside: At the end of the article they have a "Bonus: Speeding things up" section where they automate adding 300~ coupons via 300 HTTP connections in 5 seconds (instead of 60~ seconds). In my opinion if you're going to automate stuff like this, you should do so with the goal of minimizing disruption (and frankly, detection). They run the script automatically at midnight in the background via cron, so why was going slower problematic? 300 requests in a span of 5 seconds seems much more likely to trigger an IDS[0], get flagged as unusual in the logs, or similar than 300 over 60 seconds. Particularly at midnight. I'd be trying to look at human as possible and not set off automated security systems. Heck you could add a randomized delay (e.g. 1~2 seconds) between requests and it would still be completed inside of 10 minutes. Plus then nobody can reasonably accuse you of trying to "DoS" them/violate the CFAA. [0] https://en.wikipedia.org/wiki/Intrusion_detection_system https://en.wikipedia.org/wiki/Intrusion_detection_system

jonluca 7y ago
I agree - I actually built the thread pool stuff for work while I was testing, since I had the code from a previous project. If I was more worried about Safeway catching on I'd probably do something as you suggested (at the very least I'd add user agent headers and the other cookies expected from a real session, as right now it's trivial to detect my requests). Sneaker bots do this exceedingly well - it's a constant cat and mouse game to make the requests look as human as possible, very interesting space right now.
- penagwin 7y ago
  I highly recommend not making more then say 5req/s - and preferably do it at like 2am. These projects are fun IMO, but it's best to not hammer anyone's servers if it isn't necessary.
  - Scoundreller 7y ago
    But at 2AM your traffic will create a more noticeable spike... It doesn’t blend in as well...
    - penagwin 7y ago
      In my experience, unless they have excellent monitoring and a team looking to ban you they won't notice. It's half courtesy, and the other half is because it's not likely to affect their business operations (by slamming the servers when they're busy) they're less likely to notice or care.
penagwin 7y ago
Yeah from my experience they'll run into problems real fast making 300req/s in 5 seconds. It's so "noisy" and potentially disruptive. I'm honestly surprised they weren't rate limited. I mean, your SPA would have to be really messed up to make more than a handful of requests a second (even then why aren't you using sockets?) - So it's super reasonable to say that if anyone is making over 100req/s then maybe they should take an hour timeout. If you don't want to get caught doing this then you'll want a randomized delay like you said, and preferably a pool of IP addresses you proxy through. This is to prevent the automated stuff from catching you though - a human can still look at your account and wonder why you have every coupon activated 24/7.
logifail 7y ago
> you should do so with the goal of minimizing disruption (and frankly, detection) Q: If you're worried about detection, why would one blog about it/post it on HN? Isn't the fastest way to shutdown a loophole to make it public?
samstave 7y ago
Totally, and plus, it runs at midnight, and people aren’t going to be shopping so early in the morning... so span it over 6 hours. At least.
- bigwavedave 7y ago
  While I agree with spanning it over six+ hours, you might be surprised at the activity level in shopping centers at midnight. Once upon a time, I worked nightshift at walmart stocking shelves- on my days off, I'd still have to be awake all night to maintain a sleep schedule. To retain my sanity, I'd shop at other grocery stores so it didn't feel like I lived at walmart, and I was pretty surprised by how many shoppers were out and about at 1 and 2 am at stores like Ridley's or Smith's.

jonluca 7y ago

I agree - I actually built the thread pool stuff for work while I was testing, since I had the code from a previous project. If I was more worried about Safeway catching on I'd probably do something as you suggested (at the very least I'd add user agent headers and the other cookies expected from a real session, as right now it's trivial to detect my requests). Sneaker bots do this exceedingly well - it's a constant cat and mouse game to make the requests look as human as possible, very interesting space right now.

penagwin 7y ago
I highly recommend not making more then say 5req/s - and preferably do it at like 2am. These projects are fun IMO, but it's best to not hammer anyone's servers if it isn't necessary.
- Scoundreller 7y ago
  But at 2AM your traffic will create a more noticeable spike... It doesn’t blend in as well...
  - penagwin 7y ago
    In my experience, unless they have excellent monitoring and a team looking to ban you they won't notice. It's half courtesy, and the other half is because it's not likely to affect their business operations (by slamming the servers when they're busy) they're less likely to notice or care.

penagwin 7y ago

Yeah from my experience they'll run into problems real fast making 300req/s in 5 seconds. It's so "noisy" and potentially disruptive. I'm honestly surprised they weren't rate limited. I mean, your SPA would have to be really messed up to make more than a handful of requests a second (even then why aren't you using sockets?) - So it's super reasonable to say that if anyone is making over 100req/s then maybe they should take an hour timeout. If you don't want to get caught doing this then you'll want a randomized delay like you said, and preferably a pool of IP addresses you proxy through. This is to prevent the automated stuff from catching you though - a human can still look at your account and wonder why you have every coupon activated 24/7.

logifail 7y ago

> you should do so with the goal of minimizing disruption (and frankly, detection) Q: If you're worried about detection, why would one blog about it/post it on HN? Isn't the fastest way to shutdown a loophole to make it public?

samstave 7y ago

Totally, and plus, it runs at midnight, and people aren’t going to be shopping so early in the morning... so span it over 6 hours. At least.

bigwavedave 7y ago
While I agree with spanning it over six+ hours, you might be surprised at the activity level in shopping centers at midnight. Once upon a time, I worked nightshift at walmart stocking shelves- on my days off, I'd still have to be awake all night to maintain a sleep schedule. To retain my sanity, I'd shop at other grocery stores so it didn't feel like I lived at walmart, and I was pretty surprised by how many shoppers were out and about at 1 and 2 am at stores like Ridley's or Smith's.